Senior Data Engineer

Job Summary

The Senior Data Engineer is responsible for designing, developing, and maintaining scalable data infrastructure, data pipelines, and cloud-based data platforms to support analytics, business intelligence, reporting, and machine learning initiatives. The role ensures that high-quality, secure, and reliable data is efficiently collected, transformed, stored, and made accessible across the organization. Working closely with data scientists, analysts, and business stakeholders, the Senior Data Engineer enables data-driven decision-making through modern data engineering practices, robust architecture, and continuous optimization of data systems.

Key Responsibilities

Data Pipeline Development and ETL/ELT Management

• Design, develop, implement, and maintain scalable data pipelines and ETL/ELT processes.
• Leverage technologies such as Apache Airflow, dbt, Kafka, Spark, and cloud-native ETL tools including AWS Glue, Azure Data Factory, and Google Cloud Dataflow.
• Ensure data pipelines are reliable, fault-tolerant, and optimized for performance.
• Automate data ingestion, transformation, validation, and loading processes.

Data Architecture and Data Warehousing

• Design, develop, and maintain enterprise data models and data warehouse architectures.
• Build and manage modern cloud data warehouse platforms such as Snowflake, Amazon Redshift, Google BigQuery, and Azure Synapse Analytics.
• Implement dimensional modeling, data vault methodologies, and best practices for data storage and retrieval.
• Optimize database performance and data storage for scalability and efficiency.

Data Quality, Governance, and Compliance

• Ensure data quality, integrity, consistency, and reliability across all data platforms.
• Develop data quality validation rules, monitoring processes, and anomaly detection frameworks.
• Support data governance initiatives through metadata management, data lineage documentation, and regulatory compliance.
• Implement data security measures, access controls, and privacy standards across data environments.

Collaboration and Business Enablement

• Collaborate with data scientists, analysts, software engineers, and business stakeholders to understand data requirements.
• Design data solutions that support analytics, reporting, machine learning, and self-service business intelligence.
• Translate business requirements into scalable and efficient data engineering solutions.
• Support cross-functional teams by providing reliable and accessible data resources.

System Performance and Operational Excellence

• Monitor data pipeline performance and infrastructure health using observability tools such as Datadog, Prometheus, and OpenTelemetry.
• Identify, troubleshoot, and resolve production issues affecting data availability and performance.
• Continuously optimize data processing workflows to improve efficiency and reduce operational costs.
• Ensure high availability and reliability of data platforms.

DevOps and Continuous Improvement

• Implement best practices for coding standards, version control, CI/CD pipelines, and infrastructure automation.
• Utilize tools such as Git, Terraform, Kubernetes, and other DevOps technologies to support data infrastructure.
• Research, evaluate, and adopt emerging data engineering technologies, frameworks, and cloud-native solutions.
• Contribute to continuous improvement initiatives that enhance scalability, performance, and operational efficiency.

Education

• Bachelor’s Degree in Computer Science, Engineering, Information Technology, or a related field.

Experience

• Minimum of 4 years of professional experience in Data Engineering or a related field.
• Hands-on experience designing and maintaining enterprise-scale data platforms and pipelines.

Technical Skills

• Advanced proficiency in SQL, database optimization, and data modeling.
• Strong knowledge of ETL/ELT development, orchestration, and workflow automation.
• Experience with big data technologies including Apache Spark and Apache Kafka.
• Proficiency in Python, Scala, Java, or other programming languages used for data engineering.
• Hands-on experience with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
• Strong understanding of cloud data warehouse and lakehouse architectures.
• Experience with DevOps practices, CI/CD pipelines, Git, Terraform, and Kubernetes.
• Knowledge of data governance, metadata management, data lineage, security, and privacy standards.
• Ability to design scalable, high-performance data pipelines for large and complex datasets.

Behavioral Competencies

• Strong analytical and problem-solving skills.
• Excellent communication and collaboration skills.
• Ability to translate business requirements into technical data solutions.
• Strong organizational and time management skills.
• Attention to detail and commitment to data quality.
• Ability to work independently and collaboratively within cross-functional teams.
• Continuous learning mindset with a passion for emerging technologies and innovation.

Skills Required:

• Computer / Software / It / Data