Posts

Showing posts with the label file system

Featured Post

14 Top Data Pipeline Key Terms Explained

-
Image
 Here are some key terms commonly used in data pipelines 1. Data Sources Definition: Points where data originates (e.g., databases, APIs, files, IoT devices). Examples: Relational databases (PostgreSQL, MySQL), APIs, cloud storage (S3), streaming data (Kafka), and on-premise systems. 2. Data Ingestion Definition: The process of importing or collecting raw data from various sources into a system for processing or storage. Methods: Batch ingestion, real-time/streaming ingestion. 3. Data Transformation Definition: Modifying, cleaning, or enriching data to make it usable for analysis or storage. Examples: Data cleaning (removing duplicates, fixing missing values). Data enrichment (joining with other data sources). ETL (Extract, Transform, Load). ELT (Extract, Load, Transform). 4. Data Storage Definition: Locations where data is stored after ingestion and transformation. Types: Data Lakes: Store raw, unstructured, or semi-structured data (e.g., S3, Azure Data Lake). Data Warehous...
Post a Comment
Read more

Hadoop fs (File System) Commands List

-
Image
Hadoop HDSF File system commands given in this post. These are useful for your projects and interviews. HDFS commands HDFS File System Commands. Hadoop fs -cmd <args> cmd is a specific command and arg is the variable name.  The List of Commands cat  Hadoop fs –cat FILE [FILE …]  Displays the files' content. For reading compressed files.  chgrp  Hadoop fs –chgrp [-R] GROUP PATH [PATH …]  Changes the group association for files and directories. The – R option applies the change recursively.  The user must be the files' owner or a superuser.  chmod  Hadoop fs –chmod [-R] MODE[,MODE …] PATH [PATH …]  Changes the permissions of files and directories. Like, its Unix equivalent, MODE can be a 3-digit octal mode, or {augo}+/-{rwxX}. The -R option applies the change recursively. The user must be the files' owner or a superuser.  chown  Hadoop fs –chown [-R] [OWNER][:[GROUP]] PATH [PATH…]  Changes the ownership of files and di...
1 comment
Read more