Posts

Showing posts with the label file

Featured Post

14 Top Data Pipeline Key Terms Explained

-
Image
 Here are some key terms commonly used in data pipelines 1. Data Sources Definition: Points where data originates (e.g., databases, APIs, files, IoT devices). Examples: Relational databases (PostgreSQL, MySQL), APIs, cloud storage (S3), streaming data (Kafka), and on-premise systems. 2. Data Ingestion Definition: The process of importing or collecting raw data from various sources into a system for processing or storage. Methods: Batch ingestion, real-time/streaming ingestion. 3. Data Transformation Definition: Modifying, cleaning, or enriching data to make it usable for analysis or storage. Examples: Data cleaning (removing duplicates, fixing missing values). Data enrichment (joining with other data sources). ETL (Extract, Transform, Load). ELT (Extract, Load, Transform). 4. Data Storage Definition: Locations where data is stored after ingestion and transformation. Types: Data Lakes: Store raw, unstructured, or semi-structured data (e.g., S3, Azure Data Lake). Data Warehous...
Post a Comment
Read more

5 Python File Modes You Need

-
Image
Here're top five Python file modes explained. The purpose is to open, read, write the files. There are occasions you need to deal with data, which is present in the files. You need to give correct file-modes to handle the files in Python. Python file open modes 5 Python File Modes You Need Here's an example code how to you can use file mode:   filename = input ( 'Enter a filename : ' ) f1 = open (filename, 'mode' ) 1- Python File mode w   To open the file for writing, you need 'w' mode. The beauty of this is If the file does not exist, it creates one. This mode's purpose is to write the file. If you try to read, you will get an error. 2- Python File mode w+ To Open the file for Reading and Writing, you need 'w+' mode. For instance, you used w+, you have tried to read the file - after writing, it displays blank. The reason is after writing cursor position will point at the end of the file. 3- Python File mode a It appends the records at the...
Post a Comment
Read more

The best helpful HDFS File System Commands (2 of 4)

-
Image
#Top-Selected-HDFS-file-system-commands CopyFrom Local Works similarly to the put command, except that the source is restricted to a local file reference. hdfs dfs -copyFromLocal URI hdfs dfs -copyFromLocal input/docs/data2.txt hdfs://localhost/user/rosemary/data2.txt HDFS Commands Part-1of 4 copyToLocal Works similarly to the get command, except that the destination is restricted to a local file reference. hdfs dfs -copyToLocal [-ignorecrc] [-crc] URI hdfs dfs -copyToLocal data2.txt data2.copy.txt count Counts the number of directories, files, and bytes under the paths that match the specified file pattern. hdfs dfs -count [-q] hdfs dfs -count hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2 cp Copies one or more files from a specified source to a specified destination. If you specify multiple sources, the specified destination must be a directory. hdfs dfs -cp URI [URI …] hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir du Disp...
Post a Comment
Read more

The best helpful hdfs file system commands (1 of 4)

-
Image
#The best helpful hdfs file system commands: cat hadoop fs -cat FILE [ ... ] Displays the file content. For reading compressed files, you should use the TEXT command instead. chgrp hadoop fs -chgrp [-R] GROUP PATH [ PATH....] Changes the group association for files and directories. The -R option applies the change recursively.
Post a Comment
Read more