|
Search Jobvertise Jobs
| |
Jobvertise
|
Lead Databricks Engineer
Location: IN-Other Karnataka
Jobcode: indigojobs_650386
Email Job
| Report Job
Job DescriptionKey Responsibilities: Architect Lakehouse Solutions: Design, implement, and architect Lakehouse solutions on Databricks, leveraging Delta Lake, and utilize Azure services to enhance data storage and processing. In-depth Databricks Expertise: Demonstrate a deep understanding of Databricks platform features, including Spark SQL, Delta Lake, and Databricks notebooks, to optimize data engineering processes. Data Transformation: Implement advanced data transformations and quality checks within the Lakehouse architecture to ensure data accuracy, completeness, and consistency. Performance Optimization: Architect data pipelines for performance and scalability, making efficient use of Databricks clusters and Azure resources. Data Modelling: Create and implement advanced data models and schemas on Databricks, aligning with Lakehouse principles for analytical and reporting needs. Data Integration: Seamlessly integrate data from diverse sources, aligning with Lakehouse. principles for data ingestion and storage, leveraging Azure Data Lake Storage and Azure SQL Data Warehouse. Lakehouse Architecture Experience: Apply architecture principles to build and maintain a scalable, efficient, and resilient Lakehouse architecture on Databricks while utilizing Azure for enhanced data processing and storage. Data Security: Implement and maintain comprehensive data security and access controls within the Lakehouse architecture, utilizing Azure security features to safeguard sensitive data. Data Migration Expertise: Possess a deep understanding of data migration strategies and practices, particularly in Azure environments, for seamless and secure data transfers. Experience on Hadoop ecosystem will be given advantage. Collaboration: Collaborate closely with data scientists, data analysts, and cross-functional teams to understand data requirements and deliver Lakehouse-based solutions that meet their needs, incorporating Azure-based solutions and data migration strategies as needed. Documentation: Create detailed documentation for data pipelines, data models, Lakehouse architecture configurations, Azure integrations, and data migration plans. Troubleshooting: Proactively identify and resolve complex data pipeline and architecture issues, ensuring data integrity and availability, with a focus on Azure monitoring and diagnostics. Performance Monitoring: Employ advanced monitoring techniques and Azure-based tools to maintain the performance and health of the Lakehouse architecture, taking proactive measures to address potential bottlenecks or issues. Should also have understanding of cluster monitoring and sizing. Data Governance: Ensure strict adherence to data governance and data management best practices within the Lakehouse architecture, utilizing Azure-based data governance solutions. Continuous Learning: Stay updated with the latest trends and developments in Databricks, Lakehouse concepts, Azure, data migration strategies, and data engineering best practices, and incorporate them into projects. Qualifications and Skills: Bachelor's or higher degree in computer science, data engineering, or a related field. Proven experience as a data engineer with a strong focus on Databricks, Lakehouse architecture, advanced data engineering concepts, Azure services, and data migration strategies. Exceptional understanding of data engineering fundamentals, including data modeling, ETL, and data warehousing. Proficiency in programming languages such as Python, Scala, or Java. Extensive experience with Databricks platform features, advanced Delta Lake concepts, and a strong working knowledge of Azure services. Strong problem-solving skills and the ability to work in a fast-paced, collaborative environment. Excellent communication and teamwork skills. Relevant certifications in Databricks, Lakehouse, Azure services, and data migration are highly desirable. Join our team and be part of a culture that values collaboration, innovation, and excellenceEmployement Category:Employement Type: Full timeIndustry: E-Learning Role Category: E-Learning Functional Area: Not ApplicableRole/Responsibilies: Lead Databricks Engineer
Exasoft
|
