HCLTech→
Senior Technical Lead at HCLTech in City of London, England, GB
ExperiencedOn-siteFull-timeCity of London, England, GB
Skills
apache spark structured streamingdatabricks auto loaderpysparkdatabricks sqldelta lakedata modelingsap aemunity catalog
Job Description
Job Summary
Role:
- Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
- Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
- Carry out data modelling in Databricks.
- Expose the Delta tables to the visualisation layer.
- Ensure that the data pipelines are optimised.
Skillsets:
- Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
- Fluent in PySpark and Databricks SQL.
- Proven experience building Databricks architecture patterns.
Familiarity with Unity Catalog for data governance and access control.
Key Responsibilities
Role:
- Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
- Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
- Carry out data modelling in Databricks.
- Expose the Delta tables to the visualisation layer.
- Ensure that the data pipelines are optimised.
Skillsets:
- Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
- Fluent in PySpark and Databricks SQL.
- Proven experience building Databricks architecture patterns.
Familiarity with Unity Catalog for data governance and access control.
Skill Requirements
Role:
- Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
- Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
- Carry out data modelling in Databricks.
- Expose the Delta tables to the visualisation layer.
- Ensure that the data pipelines are optimised.
Skillsets:
- Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
- Fluent in PySpark and Databricks SQL.
- Proven experience building Databricks architecture patterns.
Familiarity with Unity Catalog for data governance and access control.
Other Requirements
Role:
- Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
- Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
- Carry out data modelling in Databricks.
- Expose the Delta tables to the visualisation layer.
- Ensure that the data pipelines are optimised.
Skillsets:
- Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
- Fluent in PySpark and Databricks SQL.
- Proven experience building Databricks architecture patterns.
Familiarity with Unity Catalog for data governance and access control.