Big Data Developer

Big Data Developer

14 Feb
|
IRIS Software Pvt Ltd
|
NCT

14 Feb

IRIS Software Pvt Ltd

NCT

Job Description



Big Data Developer - ( 20000059 )



Description





Job Description:





We are looking for a Spark developer who knows how to fully exploit the potential of our Spark cluster.



You will clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data to our feature developers and business analysts.



This involves both ad-hoc requests as well as data pipelines that are embedded in our production environment.



Knowledge of Java and Python is Preferred.





Responsibilities:





Create Scala/Spark jobs for data transformation and aggregation



Produce unit tests for Spark transformations and helper methods









Write Scaladoc-style documentation with all code



Design data processing pipelines





Skills:





Scala ,Scalatest, JUnit, Mockito



Apache Spark 2.x



Apache Spark RDD API



Apache Spark SQL DataFrame API



Apache Spark MLlib API



Apache Spark GraphX API



Apache Spark Streaming API



Spark query tuning and performance optimization



SQL database integration Microsoft, Oracle, Postgres, and/or MySQL



Experience working with HDFS, S3, and/or MangoDB, middlewares like MQ solace



Deep understanding of distributed systems (e.g. CAP theorem, partitioning, replication, consistency, and consensus)





Qualifications



BCA/MCA/BTECH/MTECH



Primary Location : INDIA





Organization : Iris Software



Schedule : Regular



Shift : Standard





Job Type : Full-time