Data Platform Engineer

Job Summary

The Data Platform Engineer is responsible for designing, building, operating, and optimizing the enterprise data platform that enables analytics, AI, reporting, and automation across the organization. The role focuses on scalable data pipelines, lakehouse and warehouse environments, integration frameworks, platform reliability, security, and performance. This position ensures enterprise data platforms are governed, resilient, and capable of supporting real-time and batch workloads for business users, analytics teams, AI teams, and automation functions.

Job Responsibilities 1

1. Data Platform Architecture & Infrastructure

• Design, build, and maintain enterprise data platforms including data lakes, lakehouses, and data warehouses.
• Manage cloud-based data environments such as Databricks, Snowflake, BigQuery, Synapse, or equivalent platforms.
• Ensure scalability, reliability, resilience, and security of platform infrastructure across environments.

2. Data Pipelines & Integration Engineering

• Build and maintain batch and streaming ingestion frameworks for structured and unstructured data sources.
• Develop reusable ETL / ELT frameworks, templates, and standards for enterprise data movement.
• Integrate data from ERP, CRM, APIs, flat files, operational systems, and streaming sources.

3. Workflow Automation & DevOps

• Implement orchestration tools such as Airflow, ADF, Dagster, or similar workflow schedulers.
• Establish CI/CD pipelines for data engineering deployments and automated testing processes.
• Promote infrastructure-as-code, containerization, and automation best practices.

4. Data Governance & Security Enablement

• Implement access controls, encryption, and platform security standards for enterprise data assets.
• Enable metadata management, lineage capture, data cataloging, and discovery capabilities.
• Support governance frameworks, compliance requirements, and data quality monitoring controls.

5. Performance Optimization & Reliability

• Monitor platform performance, workload usage, storage efficiency, and query responsiveness.
• Optimize compute utilization, pipeline execution times, and operational costs.
• Troubleshoot platform incidents and ensure high availability with minimal disruption.

Job Responsibilities 2

6. Backup, Recovery & Environment Management

• Manage Dev, Test, and Production environments with controlled release practices.
• Implement backup, disaster recovery, retention, and recovery procedures for critical data assets.
• Ensure continuity planning for data platform operations and service resilience.

7. Support for Analytics, AI & Automation

• Deliver curated datasets and optimized data structures for reporting, dashboards, AI models, and automation use cases.
• Enable feature-ready datasets for machine learning and predictive analytics workloads.
• Support both real-time and scheduled consumption models for enterprise users.

8. Continuous Improvement & Stakeholder Collaboration

• Work closely with architects, analysts, AI teams, automation teams, and business stakeholders to improve platform capabilities.
• Recommend new technologies and engineering practices that enhance speed, scalability, and value delivery.
• Contribute to platform roadmaps, standards, and long-term modernization initiatives.

Additional Responsibilities 3

Job Knowledge & Skills

Modern Data Architectures: Strong knowledge of lakehouse, warehouse, medallion, and scalable data platform models.

Data Engineering Tools: Expertise in ETL / ELT, orchestration, SQL, Python, Spark, and distributed processing.

Governance Foundations: Understanding of metadata, lineage, catalogs, security, and quality frameworks.

Performance Optimization: Strong capability in tuning pipelines, queries, storage, and compute efficiency.

Stakeholder Service Mindset: Ability to deliver reliable platform services to multiple business consumers.

Job Experience

Data Engineering Expertise: Minimum 5–10 years of experience in Data Engineering, Data Platform Engineering, Big Data, or Data Warehousing roles.

Enterprise Platforms: Proven experience building cloud data platforms, pipelines, and integrated data ecosystems.

Cross Functional Delivery: Experience supporting analytics, AI, automation, and enterprise reporting functions.

Competencies

Education