HiringCafe
Posted 2mo ago

Sr. Pyspark Data Engineer

@ Dataeconomy
View All Jobs
Hyderabad, Telangana, India
OnsiteFull Time
Responsibilities:Design Spark-based execution frameworks, Build reusable Spark components, Optimize Spark performance
Requirements Summary:4+ years of Apache Spark experience; PySpark preferred; Spark internals; Iceberg/Delta Lake; SQL; batch and streaming pipelines.
Technical Tools Mentioned:Apache Spark, PySpark, Spark SQL, Iceberg, Delta Lake, Kafka, JDBC, Snowflake, Parquet
Save
Mark Applied
Hide Job
Report & Hide
Job Description

Job Title: PySpark Data Engineer
Experience: 4+ Years
Location: Hyderabad

 

Job Summary:

We are seeking a Senior Spark Engineer to design and implement high-performance Spark execution patterns inside xFlows, supporting batch and streaming pipelines with built-in data quality, observability, and governance.



Requirements

Key Responsibilities

  • Design and implement Spark-based execution frameworks for xFlows pipelines.
  • Build reusable Spark components for:
    • Readers (JDBC, Files, Kafka, CDC)
    • Transformers (Join, Filter, Aggregate, Window, Union)
    • Writers (Iceberg, Delta, Parquet, Snowflake)
  • Optimize Spark performance (partitioning, caching, shuffles, memory).
  • Implement Data Quality & Reconciliation execution patterns.
  • Handle schema evolution, CDC, watermarking, and checkpoints.
  • Integrate Spark jobs with EMR Serverless / Data bricks / Kubernetes.
  • Publish execution metrics, logs, and lineage for observability.
  • Work closely with platform & UI teams to support no-code execution.

Required Skills

  • 4+ years of experience with Apache Spark
  • Strong expertise in PySpark (preferred) or Spark Scala
  • Deep understanding of Spark internals (DAGs, stages, shuffles, caching)
  • Experience with Iceberg / Delta Lake
  • Strong SQL skills
  • Experience with batch and streaming pipeline


Benefits

  • Comprehensive Medical Coverage:  
    Health insurance of INR 7.0 Lakhs for you and your family (up to 6 members), ensuring complete peace of mind.
  • Robust Protection Plans:  
    Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones.
  • Retirement Benefits: 
    PF and Gratuity provided as per standard government regulations.
  • Flexible Work Options: 
    Enjoy hybrid work arrangements & flexible working hours
  • Generous Leave Policy:  
    21 days of annual leave, in addition to 10 company-declared holidays.
  • Employee Well-being Spaces: 
    Access to a dedicated break-out area with round-the-clock refreshments for relaxation and rejuvenation.