Posts

Showing posts with the label PySpark

Featured Post

Python Set Operations Explained: From Theory to Real-Time Applications

Image
A  set  in Python is an unordered collection of unique elements. It is useful when storing distinct values and performing operations like union, intersection, or difference. Real-Time Example: Removing Duplicate Customer Emails in a Marketing Campaign Imagine you are working on an email marketing campaign for your company. You have a list of customer emails, but some are duplicated. Using a set , you can remove duplicates efficiently before sending emails. Code Example: # List of customer emails (some duplicates) customer_emails = [ "alice@example.com" , "bob@example.com" , "charlie@example.com" , "alice@example.com" , "david@example.com" , "bob@example.com" ] # Convert list to a set to remove duplicates unique_emails = set (customer_emails) # Convert back to a list (if needed) unique_email_list = list (unique_emails) # Print the unique emails print ( "Unique customer emails:" , unique_email_list) Ou...

AWS CLI PySpark a Beginner's Comprehensive Guide

Image
AWS (Amazon Web Services) and PySpark are separate technologies, but they can be used together for certain purposes. Let me provide you with a beginner's guide for both AWS and PySpark separately. AWS (Amazon Web Services): Amazon Web Services (AWS) is a cloud computing platform that offers a wide range of services for computing power, storage, databases, machine learning, analytics, and more. 1. Create an AWS Account: Go to the AWS homepage. Click on "Create an AWS Account" and follow the instructions. 2. Set Up AWS CLI: Install the AWS Command Line Interface (AWS CLI) on your local machine. Configure it with your AWS credentials using AWS configure. 3. Explore AWS Services: AWS provides a variety of services. Familiarize yourself with core services like EC2 (Elastic Compute Cloud), S3 (Simple Storage Service), and IAM (Identity and Access Management). PySpark: PySpark is the Python API for Apache Spark, a fast and general-purpose cluster computing system. It allows you ...

How to Handle Spaces in PySpark Dataframe Column

Image
In PySpark, you can employ SQL queries by importing your CSV file data to a DataFrame. However, you might face problems when dealing with spaces in column names of the DataFrame. Fortunately, there is a solution available to resolve this issue. Reading CSV file to Dataframe Here is the PySpark code for reading CSV files and writing to a DataFrame. #initiate session spark = SparkSession.builder \ .appName("PySpark Tutorial") \ .getOrCreate() #Read CSV file to df dataframe data_path = '/content/Test1.csv' df = spark.read.csv(data_path, header=True, inferSchema=True) #Create a Temporary view for the DataFrame df2.createOrReplaceTempView("temp_table") #Read data from the temporary view spark.sql("select * from temp_table").show() Output --------+-----+---------------+---+ |Student| Year|Semester1|Semester2| | ID | | Marks | Marks | +----------+-----+---------------+ | si1 |year1|62.08| 62.4| | si1 |year2|75.94| 76.75| | si...