Big Data Developer

Big Data Developer

14 Feb
IRIS Software Pvt Ltd

14 Feb

IRIS Software Pvt Ltd


Job Description

Big Data Developer - ( 20000059 )


Job Description:

We are looking for a Spark developer who knows how to fully exploit the potential of our Spark cluster.

You will clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data to our feature developers and business analysts.

This involves both ad-hoc requests as well as data pipelines that are embedded in our production environment.

Knowledge of Java and Python is Preferred.


Create Scala/Spark jobs for data transformation and aggregation

Produce unit tests for Spark transformations and helper methods

Write Scaladoc-style documentation with all code

Design data processing pipelines


Scala ,Scalatest, JUnit, Mockito

Apache Spark 2.x

Apache Spark RDD API

Apache Spark SQL DataFrame API

Apache Spark MLlib API

Apache Spark GraphX API

Apache Spark Streaming API

Spark query tuning and performance optimization

SQL database integration Microsoft, Oracle, Postgres, and/or MySQL

Experience working with HDFS, S3, and/or MangoDB, middlewares like MQ solace

Deep understanding of distributed systems (e.g. CAP theorem, partitioning, replication, consistency, and consensus)



Primary Location : INDIA

Organization : Iris Software

Schedule : Regular

Shift : Standard

Job Type : Full-time