Posts

Showing posts with the label List duplicates

Featured Post

14 Top Data Pipeline Key Terms Explained

-
Image
 Here are some key terms commonly used in data pipelines 1. Data Sources Definition: Points where data originates (e.g., databases, APIs, files, IoT devices). Examples: Relational databases (PostgreSQL, MySQL), APIs, cloud storage (S3), streaming data (Kafka), and on-premise systems. 2. Data Ingestion Definition: The process of importing or collecting raw data from various sources into a system for processing or storage. Methods: Batch ingestion, real-time/streaming ingestion. 3. Data Transformation Definition: Modifying, cleaning, or enriching data to make it usable for analysis or storage. Examples: Data cleaning (removing duplicates, fixing missing values). Data enrichment (joining with other data sources). ETL (Extract, Transform, Load). ELT (Extract, Load, Transform). 4. Data Storage Definition: Locations where data is stored after ingestion and transformation. Types: Data Lakes: Store raw, unstructured, or semi-structured data (e.g., S3, Azure Data Lake). Data Warehous...
Post a Comment
Read more

Python Delete Duplicates in List Faster Way

-
Image
Removing duplicates in List simplified using SET method. It's a simple method. Just you need SET and Print to remove duplicates. Removing duplicates is common in Data science projects. What is list A list is a collection of elements. The elements can be duplicates or non-duplicates. Today's task is to remove duplicate elements in the List. Faster way to remove list duplicates Create a List Use SET Print the result List with duplicates my_list = ['The', 'unanimous', 'Declaration', 'of', 'the', 'thirteen','united', 'States', 'of', 'America,', 'When', 'in', 'the', 'Course', 'of', 'human'] Apply set method >>> non_dupes = set(my_list) Print Final list >>> print(non_dupes) Here, if you observe, there are no duplicates. The duplicates are now removed. It displays only non-duplicate values.  Here 'the' is a duplicate value. That'...
1 comment
Read more