Report this job

---

What is the problem? *

Incorrect company
Incorrect location
Job is expired
Job may be a scam
Other

Job Description: Spark Streaming Developer - RDD

Role: Spark Streaming RDD and Spark SQL

We are seeking a highly skilled and experienced Spark Streaming Developer with expertise in RDD (Resilient Distributed Datasets) and Spark SQL. As a Spark Streaming Developer, you will be responsible for real-time and batch data processing, leveraging Spark Streaming and Spark SQL technologies. You will play a critical role in developing scalable and high-performance data processing solutions.

Responsibilities:

• Develop and maintain Spark Streaming applications using RDD (Resilient Distributed Datasets) for real-time data processing.
• Design, develop, and optimize Spark SQL queries for efficient data manipulation and analysis.
• Collaborate with cross-functional teams to gather requirements, understand data processing needs, and design appropriate solutions.
• Work with large-scale datasets and implement data processing pipelines for both streaming and batch data.
• Optimize Spark applications for performance and scalability, ensuring efficient data processing and minimal latency.
• Provide expertise and support in maintaining resiliency in Spark applications, including fault tolerance and data recovery mechanisms.
• Monitor Spark applications, identify and troubleshoot performance bottlenecks, and implement optimizations.
• Stay up to date with the latest advancements in Spark Streaming and Spark SQL technologies, and actively contribute to the improvement of data processing solutions.

Requirements:

• Minimum 8 years of experience as a Spark Streaming Developer, with a strong focus on RDD and Spark SQL.
• Deep understanding and hands-on experience with Spark Streaming and RDD (Resilient Distributed Datasets) for real-time data processing.
• Proficiency in Spark SQL for data manipulation, querying, and analysis.
• Strong experience in developing and optimizing Spark applications for performance and scalability.
• Knowledge of data streaming concepts, such as micro-batching and windowing.
• Experience with large-scale data processing using Spark, Hadoop, or similar technologies.
• Strong understanding of data processing architectures, data pipelines, and ETL (Extract, Transform, Load) processes.
• Familiarity with resiliency support in Spark, including fault tolerance and data recovery mechanisms.
• Excellent problem-solving skills and the ability to troubleshoot and optimize data processing pipelines.
• Proven ability to work effectively in a collaborative, cross-functional team environment.
• Strong communication and interpersonal skills, with the ability to convey complex technical concepts to both technical and non-technical stakeholders.

If you are a highly skilled Spark Streaming Developer with expertise in RDD and Spark SQL, and you are passionate about real-time and batch data processing, we invite you to join our team. Help us build scalable and efficient data processing solutions using the power of Spark Streaming and Spark SQL.