Posts

Showing posts with the label data engineer career path

Featured Post

How to Build CI/CD Pipeline: GitHub to AWS

Image
 Creating a CI/CD pipeline to deploy a project from GitHub to AWS can be done using various AWS services like AWS CodePipeline, AWS CodeBuild, and optionally AWS CodeDeploy or Amazon ECS for application deployment. Below is a high-level guide on how to set up a basic GitHub to AWS pipeline: Prerequisites AWS Account : Ensure access to the AWS account with the necessary permissions. GitHub Repository : Have your application code hosted on GitHub. IAM Roles : Create necessary IAM roles with permissions to interact with AWS services (e.g., CodePipeline, CodeBuild, S3, ECS, etc.). AWS CLI : Install and configure the AWS CLI for easier management of services. Step 1: Create an S3 Bucket for Artifacts AWS CodePipeline requires an S3 bucket to store artifacts (builds, deployments, etc.). Go to the S3 service in the AWS Management Console. Create a new bucket, ensuring it has a unique name. Note the bucket name for later use. Step 2: Set Up AWS CodeBuild CodeBuild will handle the build proces

5 Essential IT Skills for Data Engineers

Image
Data engineers need the following skills. These skills help you get nice job in any analytics company. Photo Credit: Srini Five Top Skills Need Skill-1 Experience working with big data tools such as MapReduce, Pig, Spark, Kafka and NoSQL data stores such as MongoDB, Cassandra, HBase, etc. Skill-2 Expertise in multi-structured data modeling, reporting on NoSQL & structured database technologies such as HBase and Cassandra, SQL. Skill-3 Experience with languages such as Python, Perl, Ruby, Java, Scala, R etc. Skill-4 Strong data & visual presentation skills and ability to explain insights using tools like tableau, D3 charts or other tools. Skill-5 Basic knowledge and experience of statistical analysis tools such as R.