Software Engineer, Data Ingestion

Overview

In this role you will own and improve the data ingestion layer for Wayve’s autonomous driving platform. You will debug failing pipelines, handle corrupt data, and optimize Spark-based ETL workflows to scale with petabytes of real-world driving data. You’ll build resilient pipelines that isolate bad data and streamline multi-step workflows for annotation, data science, and model training. The role offers impact across data engineering, reliability, and performance, within a fast-moving AI platform focused on safe and scalable autonomous driving.

Pay / Benefits

• equity participation
• relocation support and visa sponsorship
• hybrid working with core hours
• learning and development budgets
• health insurance and dental
• maternity/paternity leave and wellbeing resources

Responsibilities

• Debug and unblock failing ingestion pipelines, stalled jobs and processing queues
• Investigate corrupt, malformed or unexpected data from diverse sources
• Optimize Spark jobs and batch-processing pipelines for throughput, reliability and compute efficiency
• Design and implement more resilient pipelines to prevent a single bad segment from blocking broader workflows
• Improve monitoring, retries and failure recovery across multi-step ingestion workflows
• Prioritize and unblock datasets for downstream annotation, data science and model-training teams and contribute to technical direction of the ingestion platform
• Work on high-volume pipelines for annotation, data science, model training and evaluation
• Develop resilient systems that isolate problematic data and orchestrate complex dependencies, queues, retries and priority data flows
• Make pragmatic improvements to performance, cost and operability of the Spark-based data platform
• Ingest varied data formats from partners, suppliers and other third-party sources

Key requirements

• Strong production experience with Apache Spark and Python
• Built, operated or debugged large-scale ingestion, ETL or distributed data-processing pipelines
• Ability to diagnose production failures involving corrupt data, stalled queues, retries and complex dependencies
• Understanding of optimizing high-throughput systems for reliability, performance and cost
• Ability to learn an existing system quickly, work independently and deliver practical improvements in a fast-moving environment
• Experience with Airflow, Flyte, Databricks, Delta Lake, Scala or Java would be an advantage

• Independent and proactive problem-solver
• Good debugging mindset and pragmatic mindset
• Ability to deliver practical improvements in a fast-moving environment
• Apache Spark
• Python
• ETL pipelines

…