Report this job

---

What is the problem? *

Incorrect company
Incorrect location
Job is expired
Job may be a scam
Other

Data & Analytics Data Engineer - SeniorThe Data & Analytics Data Engineer is responsible to support both Data & Analytics strategy and solutions across critical business areas. This role will work as a member of a multi-functional team to outline analytic opportunities and participate in solution design, development, and deployment of analytical solutions. This position will have access to and ensures protection of confidential data. Responsibilities: Retrieve, integrate and prepare data for analysis by Data & Analytics team and business user community Technical interface with IT local landscape teams, and Business user community.Roles & Responsibilities Time %-Retrieve, migrate, integrate and build data products employing industry proven disciplines of Data Acquisition, Integration, Curation and Distributed data processing to support Data & Analytics Use Cases in a Microsoft Azure based Data Warehouse. Design, deploy and automate data pipelines in an Azure DataBricks based Lakehouse using Azure Datafactory/Databricks using industry proven best practices transforming and curating datasets for rapid consumption and discovery. 30 Solid experience in building modular, extensible and highly resilient data pipelines in MS Azure (ADLS, Data Factory, Databricks, and Integration Runtime) adhering to ETL/ELT principles to enable and support BI/reporting application development using MS Power BI. Solid experience in migrating on-premise data pipelines built using technologies like Informatica to MS Azure Data Factory/Databricks. Solid experience with Microsoft Azure based Enterprise Data Warehouse implementations in the aspects of DWH/Datamart design, development and Data Engineering for supporting enterprise scale BI/Reporting applications. Solid experience with Microsoft Azure(ADLS, Data Factory, Integration Runtime, Express Route, Databricks(Lakehouse, Delta Lake, Unity catalog), Azure SQL Datawarehouse, Synapse, MS SQL Server Extensive experience with data pipeline automation, orchestration and administration using DevOps tool chain (E.g. Artifactory, GitHub, Confluence, Jira, ServiceNow and Azure Apps to support large, complex daily batch ingestion processes with emphasis on Delta processing techniques. Knowledge in migrating large, complex on-premise Data warehouses built using technologies like IBM DB2 (LUW, z/OS) to MS Azure based datawarehouses/lakehouses. Experience in providing technical expertise and mentoring others of the best practices of the data engineering space and coordinating the development and support activities. Knowledge of Data Modeling, Metadata Management techniques(e.g. data lineage) using tools like Databricks, Sybase Power Designer, Alation Data Catalog.Technical interface with Data & Analytics teams, IT local landscape teams, Global Cloud Solutions teams, Source Systems, Team Leads as well as representatives from vendors to identify and resolve issues and remove obstacles in achieving the Data & analytics goals. 20Expertise in building streamlined, efficient data pipelines and workflows to support custom workloads ensuring data quality standards using Microsoft Azure Technologies (Data Factory, ADLS, Databricks, Integration Runtime, Express Route, Power BI, DevOps, MLFlow, Artifactory, Azure SQL Datawarehouse, Synapse 20 Expertise in assembling large, complex data sets meeting the unique functional requirements of the Datawarehouse/BI applications. Transform raw and disparate data into inputs for Business Intelligence applications and organize data into clean and accessible sets. Automate data processing workflows and pipelines using CA Autosys R11, Python, Shell Scripts and Azure DevOps & CI/CD processes with structural understanding of data products and integration with APIs using JSON or xml. Experience (distributed cluster environments) with Unix based command line interface and bash scripts.Query, retrieve, integrate and prepare data for profiling, analysis and for supporting the development of BI/reporting applications. 10 Analyze/Profile various data sets to quickly identify sensitive fields and help in formulating rules for data anonymization and tokenization. Collecting descriptive statistics like min, max, count and sum. Collecting data types, length and recurring patterns. Tagging data with keywords, descriptions or categories. Performing data quality assessment, risk of performing joins on the data. Discovering metadata and assessing its accuracy. Identifying distributions, key candidates, functional dependencies, embedded value dependencies, and performing inter-table analysis.Prepare datasets conducible for consumption in Data Visualization tools (e.g. Power BI 10 Integrate DataWarehouse/Lakehouse based pipelines and workflows with Data Visualization Platforms like (Power BI) to provide multiple data streams to enable rich visualizations for business user consumption. Administer and monitor DataLake/Lakehouse batch and near real-time data processing capabilities. Establish interfaces and enable connectivity between Azure and wide variety of Source Systems including application databases, DWHs, Datamarts in the landscape and other analytics systems.Partners and consults with business user community to understand their requirements. 10Researches, recommends designs and processes to improve business application performance. Defines and implements industry standard data engineering processes, best practices aiding data governance, certification, management policies & mechanisms. Assists in maintaining business documentation for the data and data structures using tools like GitHub, Confluence and Alation. Participate in the development and communication of data strategy and roadmaps.Experience (Required) 10+ Years experience designing and building data pipelines in Microsoft Azure (Data Factory, Databricks (Lakehouse, Delta Lake, Unity catalog), ADLS, Integration Runtime, Express route & MS SQL Server) supporting Datawarehouses. Proficient in creating data pipelines and in hybrid data movement from on-prem systems to MS Azure. Experience with object oriented programing languages such as Python, R, Scala, and Java. Experience using wide variety of file formats and compression techniques. Experience with CI/CD processes.Additional Skills Knowledge of emerging Data Engineering tools & technologies. Broad understanding and experience with BI and real time analytics platforms.Analytical skills Ability to collect data, establish facts, and identify trends and variances. Ability to Integrate information from a variety of sources with varied levels of complexity Ability to Review and interpret and evaluate information. Ability to Formulate and test hypotheses for the purpose of forecasting outcomes.