Infosys Limited→
Scala, Spark/pyspark Professional at Infosys Limited in Bengaluru
ExperiencedOn-siteBengaluru
Job Description
Educational Requirements
Bachelor of Engineering, BTech, BSc, BCA, MCA, MSc, MTech
Service Line Data Analytics Unit
Responsibilities - Design and implement scalable data pipelines using Apache Spark (Scala and/or PySpark)
- Work extensively with Spark Core, Spark SQL, DataFrames, and Datasets
- Develop batch and real-time data processing solutions using Spark Streaming / Structured Streaming
- Optimize Spark jobs for performance, memory management, and parallel processing
- Develop robust and efficient applications using Scala and Python
- Write reusable, modular, and maintainable code
- Implement business logic and transformations on large datasets
- Build and maintain ETL/ELT pipelines for large-scale data ingestion and transformation
- Process structured and unstructured data from multiple sources
- Ensure data validation, quality, and consistency
- Work with file formats like Parquet, ORC, Avro, JSON, CSV
- Work with Hadoop ecosystem (HDFS, Hive, YARN)
- Integrate Spark jobs with data lakes and warehouses
- Handle large datasets with distributed computing techniques
- Work with cloud platforms (AWS/Azure/GCP) for big data solutions
- Utilize services such as AWS EMR, Glue, S3 / Azure Databricks / Synapse
- Integrate pipelines with APIs and external systems
- Collaborate with data engineers, architects, and business teams
- Lead technical discussions and provide guidance to junior developers
- Participate in code reviews and best practice implementation
- Work in Agile/Scrum environments
- Core Skills59 years of experience in data engineering / big data development
- Strong hands-on expertise in Scala (mandatory for this role)
- Extensive experience with Apache Spark (Scala and/or PySpark)
- Solid understanding of ETL processes and data pipelines
- Strong proficiency in SQL and database concepts
- Deep knowledge of Spark architecture and execution model
- Experience with Spark performance tuning and optimization
- Strong data modeling and warehousing concepts
- Familiarity with version control tools (Git)
- Understanding of distributed computing principles
- Experience with Spark Streaming / Kafka
- Hands-on with Databricks platform
- Knowledge of Airflow or workflow orchestration tools
- Familiarity with Docker/Kubernetes
- Exposure to NoSQL databases (Cassandra, MongoDB, HBase)
- Primary skills:Domain- >Finacle-Core-Functional- >Finacle-Core-WMS- >Grand Master,Technology- >Big Data - Data Processing- >Spark,Technology- >Java- >Apache
- Technology- >Java- >Apache- >Scala
- Technology- >Big Data - Data Processing- >Spark- >SparkSQL
- Technology- >Big Data - Data Processing- >PySpark