Posted 5d ago

Group Lead - Data Quality Engineer

@ DP World
Bangalore, Karnataka, India
OnsiteFull Time
Responsibilities:Data policy, Observability platform, DataApps development
Requirements Summary:8+ years in data quality engineering; strong Databricks/Azure experience; SQL and Python proficiency; UI/tooling for data platforms.
Technical Tools Mentioned:Azure, Databricks, ADLS Gen2, Spark, Streamlit, Python, Databricks Apps, REST API, Great Expectations (preferred), Deequ (preferred), Soda (preferred)
Save
Mark Applied
Hide Job
Report & Hide
Job Description

KEY ACCOUNTABILITIES

  • Data Quality Policy & Framework Implementation

  • Define and operationalize enterprise Data Quality policies, procedures, and standards.

  • Establish standardized data quality dimensions and certification frameworks.

  • Implement scalable validation frameworks across ingestion, transformation, and serving layers.

  • Embed “quality-by-design” principles into data product lifecycle.

  • Data Observability Platform Development

  • Design and implement end-to-end data observability capabilities including:

    • Data freshness and SLA monitoring

    • Volume and distribution anomaly detection

    • Schema drift and pipeline health monitoring

    • Data lineage validation and reliability tracking

    • Develop automated alerting and incident detection mechanisms.

  • Custom Data Applications (DataApps) Development

  • Build custom Data Quality and Observability applications using:

    • Databricks native capabilities

    • Streamlit / Databricks Apps

    • Python-based backend services

  • Develop user interfaces enabling:

    • Data quality rule configuration

    • Dataset certification workflows

    • Quality score visualization

    • Issue tracking and remediation workflows

    • Enable self-service quality monitoring for engineering and analytics teams.

  • Azure & Databricks Platform Integration

  • Implement data quality checks within Azure-based data pipelines and Databricks workflows.

  • Integrate monitoring with:

    • ADLS Gen2

    • Databricks Lakehouse architecture

    • Batch and streaming pipelines

    • Develop reusable frameworks leveraging Spark and Delta Lake.

    • Optimize performance and scalability of quality validation workloads.

  • Automation & Engineering Excellence

  • Integrate DQ checks into CI/CD and deployment pipelines.

  • Develop metadata-driven quality monitoring solutions.

  • Implement automated remediation and self-healing workflows where applicable.

  • Ensure auditability, traceability, and governance compliance.

  • Metrics, Reporting & Adoption

  • Define enterprise Data Quality KPIs and reliability SLAs.

  • Build dashboards tracking platform-wide data trust scores.

  • Drive adoption of standardized DQ practices across engineering teams.

  • Support audit and compliance reporting initiatives.

  • Data Quality Score

  • Leadership & Collaboration

  • Act as technical lead for Data Quality and Observability engineering.

  • Mentor engineers on best practices for data reliability.

  • Collaborate with Data Engineering, Governance, and Platform Architecture teams.

  • Contribute to long-term evolution of the enterprise data platform.

QUALIFICATIONS, EXPERIENCE AND SKILLS

Education

  • Bachelor’s or master’s degree in computer science, Data Engineering, Information Systems, or related field.

     

 

Experience

  • 8+ years of experience in Data Quality engineering roles within Data Platforms/Data Engineering teams.

  • Proven experience building custom applications on Databricks or data platforms.

  • Experience designing enterprise Data Quality or Data Observability solutions.

  • Hands-on experience developing internal data tools or platform applications.

 

Technical Skills (Required)

  • Cloud & Data Platform

     

    Strong expertise in:

  • Microsoft Azure
  • Databricks Lakehouse platform
  • ADLS Gen2
  • Distributed data processing using Spark
  • Application & DataApp Development
    • Experience building DataApps using:
      • Streamlit
      • Databricks Apps or notebook-based applications
      • Python backend development
      • Experience designing UI-driven data engineering tools or internal platforms.
    • Data Quality & Observability
      • Experience implementing data validation frameworks.
      • Strong SQL and Python programming skills.
      • Knowledge of anomaly detection, monitoring, and data reliability concepts.
    • Engineering & Integration
      • CI/CD integration for data pipelines.
      • REST API integrations and automation workflows.
      • Metadata-driven architectures and lineage concepts.

Core Competencies

  • Platform-first engineering mindset.
  • Strong problem-solving and analytical thinking.
  • Ability to translate governance requirements into scalable technical solutions.
  • Strong stakeholder collaboration and communication skills.
  • Ownership mindset with ability to lead initiatives end-to-end.

 

Preferred 

  • Experience with Great Expectations, Deequ, Soda, or similar frameworks.
  • Experience with streaming data validation.
  • Exposure to AI-driven data observability or anomaly detection.
  • Experience building enterprise internal developer platforms.

 

#LI-AA6