Posts

Showing posts with the label Sets

Featured Post

14 Top Data Pipeline Key Terms Explained

-
Image
 Here are some key terms commonly used in data pipelines 1. Data Sources Definition: Points where data originates (e.g., databases, APIs, files, IoT devices). Examples: Relational databases (PostgreSQL, MySQL), APIs, cloud storage (S3), streaming data (Kafka), and on-premise systems. 2. Data Ingestion Definition: The process of importing or collecting raw data from various sources into a system for processing or storage. Methods: Batch ingestion, real-time/streaming ingestion. 3. Data Transformation Definition: Modifying, cleaning, or enriching data to make it usable for analysis or storage. Examples: Data cleaning (removing duplicates, fixing missing values). Data enrichment (joining with other data sources). ETL (Extract, Transform, Load). ELT (Extract, Load, Transform). 4. Data Storage Definition: Locations where data is stored after ingestion and transformation. Types: Data Lakes: Store raw, unstructured, or semi-structured data (e.g., S3, Azure Data Lake). Data Warehous...
Post a Comment
Read more

Sets Vs Lists Python Programmer Tips

-
Image
Sets are only useful when trying to ensure unique items are preserved. Before sets were available, it was common to process items and check if they exist in a list (or dictionary) before adding them. List example Here unique is an empty list. Every time I compare with this list, and if it is not duplicated then the input item will append to the unique list.  >>> unique = []  >>> for name in ['srini', 'srini', 'rao', 'srini']:  ... if name not in unique:  ... unique.append(name)  ... >>> unique ['srini', 'rao'] There is no need to do this when using sets. Instead of appending you add to a set: Set example >>> for name in ['srini', 'srini', 'rao', 'srini']: ... unique.add(name)  ...  >>> unique {'srini', 'rao'} Just like tuples and lists, interacting with sets have some differences on how to access their items. You can't index them like lists an...
Post a Comment
Read more