Posts

Showing posts with the label Data-storage

Featured Post

Top Questions People Ask About Pandas, NumPy, Matplotlib & Scikit-learn — Answered!

Image
 Whether you're a beginner or brushing up on your skills, these are the real-world questions Python learners ask most about key libraries in data science. Let’s dive in! 🐍 🐼 Pandas: Data Manipulation Made Easy 1. How do I handle missing data in a DataFrame? df.fillna( 0 ) # Replace NaNs with 0 df.dropna() # Remove rows with NaNs df.isna(). sum () # Count missing values per column 2. How can I merge or join two DataFrames? pd.merge(df1, df2, on= 'id' , how= 'inner' ) # inner, left, right, outer 3. What is the difference between loc[] and iloc[] ? loc[] uses labels (e.g., column names) iloc[] uses integer positions df.loc[ 0 , 'name' ] # label-based df.iloc[ 0 , 1 ] # index-based 4. How do I group data and perform aggregation? df.groupby( 'category' )[ 'sales' ]. sum () 5. How can I convert a column to datetime format? df[ 'date' ] = pd.to_datetime(df[ 'date' ]) ...

How to Retain data in Kafka and Get Additional Time for Analysis

Image
The default topic retention period in Kafka is seven days. However, you can change the current retention period and keep data for a few more days. Hence it provides you additional time for analysis to get business insights. Kafka retention period The retention period, you can set on two parameters of bytes and time. Due to cheap storage costs, companies wish to extend the data retention period.  The retention period setup you need in the broker.  It is not a deviation that Kafka is designed only for Seven days, and why we need to change it. Since space is cheaper, we can extend the retention period. Setup for the retention period Below is the setup in the broker configuration file for the retention period. log.retention.bytes The most significant size threshold in bytes for deleting a log. log.retention.ms The length in milliseconds of a log will be maintained before being deleted. log.retention.minutes Length before deletion in minutes. log.retention.ms is used as well if bot...