Report this Job
Databricks Lead JD
Job Summary:
We are seeking an experienced Azure Databricks Lead with strong expertise in PySpark and a track record of successful project delivery. As an Azure Databricks Lead, you will be responsible for leading and managing a team of PySpark developers, designing and implementing data engineering and analytics solutions using Azure Databricks, and collaborating with stakeholders to drive business outcomes.
Key Responsibilities:
Lead and manage a team of PySpark developers, providing technical guidance, mentoring, and fostering a collaborative environment.
Design, develop, and implement end-to-end data engineering and analytics solutions using Azure Databricks.
Collaborate with business stakeholders to understand their requirements and translate them into technical designs and deliverables.
Architect and optimize data pipelines for data ingestion, transformation, processing, and storage using PySpark and other Azure services.
Develop robust and scalable PySpark code for data transformation, data cleansing, data integration, and data aggregation.
Implement and optimize data models, schemas, and data structures for efficient data processing and analysis.
Ensure data quality and data governance standards are followed throughout the data engineering process.
Collaborate with data architects, data scientists, and business analysts to develop and deploy machine learning and advanced analytics models on Azure Databricks.
Troubleshoot and resolve technical issues, performance bottlenecks, and data-related problems in Azure Databricks environments.
Stay up-to-date with the latest Azure Databricks features, tools, and technologies, and identify opportunities for process improvement and innovation.
Provide technical leadership and expertise in PySpark, Azure Databricks, and the overall data engineering domain.
Collaborate with cross-functional teams, including infrastructure, security, and operations teams, to ensure successful project delivery and operational excellence.
Requirements:
Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.
8-10 years of hands-on experience in data engineering, with a strong focus on PySpark and Azure Databricks.
Proven experience leading and managing teams in data engineering projects, preferably using Azure Databricks.
Expertise in building scalable and optimized data pipelines using PySpark and SQL.
Strong understanding of big data concepts, data lakes, and distributed computing architectures.
In-depth knowledge of Azure services related to data engineering and analytics, such as Azure Data Lake Storage, Azure SQL Data Warehouse, Azure Data Factory, and Azure Synapse Analytics.
Experience in designing and implementing ETL/ELT processes and data integration workflows.
Proficiency in programming languages such as Python and SQL.
Strong understanding of data modeling, data warehousing, and data governance principles.
Experience with cloud-based data platforms and technologies, preferably Azure.
Excellent problem-solving, analytical, and communication skills.
Relevant certifications in Azure Databricks and Azure data services are a plus.
Apptad Inc
