Senior Data Engineer & Technical Delivery Lead (Python / Databricks)

Location: London (Hybrid – 2-3 days onsite)

Contract: Initial 6 months

We're currently supporting a major Financial Services organisation that's expanding its Data Engineering function and is looking to hire multiple contractors across Senior Data Engineer and Technical Delivery Lead level.

Whether you're a highly technical Senior Data Engineer looking to remain hands-on, or an experienced Technical Lead who enjoys combining delivery leadership with coding, we'd be keen to speak with you.

You'll be joining a large-scale cloud data transformation programme, helping design, build and optimise enterprise data platforms using Python, Databricks and Spark within a modern Lakehouse environment.

Key Responsibilities

1. Design, develop and maintain scalable data pipelines using Python and Databricks.
2. Build, optimise and support enterprise ETL/ELT workflows using Apache Spark and Delta Lake.
3. Design and develop robust data models and Lakehouse architectures.
4. Implement and manage data workflows within the Databricks ecosystem.
5. Ensure high levels of data quality, governance and pipeline reliability.
6. Optimise performance across large-scale distributed data processing platforms.
7. Collaborate with architects, product owners, analysts and engineering teams to deliver high-quality data solutions.
8. Implement monitoring, logging and alerting across critical data platforms.
9. Drive engineering best practice through code reviews, mentoring and technical collaboration.
10. For Technical Delivery Lead opportunities, provide technical leadership, architectural oversight and end-to-end delivery of complex data engineering initiatives.

Required Skills & Experience

1. Strong commercial experience developing data engineering solutions using Python .
2. Extensive hands-on experience with Databricks , including Workflows, Notebooks and Delta Lake.
3. Strong experience with Apache Spark / PySpark .
4. Proven experience building scalable ETL/ELT pipelines.
5. Strong SQL and data modelling skills.
6. Experience designing modern Lakehouse architectures.
7. Experience working with cloud platforms including AWS, Azure or GCP.
8. Experience with modern data warehousing technologies such as Snowflake, Redshift or BigQuery.
9. Experience with orchestration tools such as Airflow or Databricks Workflows.
10. Previous experience working within Agile delivery environments.

Desirable Experience

1. Databricks certifications.
2. Kafka or Structured Streaming.
3. CI/CD and DevOps practices.
4. Docker and Kubernetes.
5. Exposure to Machine Learning pipelines.
6. Experience within Financial Services or other highly regulated environments.

About You

You'll be an experienced Data Engineer or Technical Lead with a passion for building modern, scalable data platforms. Comfortable working in a collaborative Agile environment, you'll enjoy solving complex technical problems and delivering high-quality engineering solutions. If you're applying at Technical Delivery Lead level, you'll also bring strong leadership, stakeholder management and mentoring experience whilst remaining hands-on technically.