Software Engineer – Autonomy Behavior ML Data Optimization Team

Candidates must be legally authorized to work in the United States and must not require any current or future employment visa sponsorship for this position. The client is looking for a Software Engineer – Autonomy Behavior ML Data Optimization Tea m to build and optimize the data pipelines and software platforms that power machine learning workflows for autonomous driving .In this role, you’ll develop scalable data pipelines, backend services, and user-facing tools that support dataset creation, embedding search, and ML data optimization. While primarily focused on backend and data engineering, you'll also contribute to frontend features that improve workflow management and user experience .The ideal candidate is a strong Python engineer who thrives in data-intensive environments and enjoys building scalable systems that support machine learning and large-scale data processing . What You'll DoDesign, build, and maintain Airflow pipelines that orchestrate end-to-end dataset creation, refresh, and management workflow s.Develop scalable Python/FastAPI services supporting embedding search, bulk operations, and dataset managemen t.Build and optimize distributed data processing pipelines using Ray for embedding generation, clustering, FAISS index creation, cache generation, and large-scale dataset processin g.Improve pipeline reliability, observability, and performance through monitoring, optimization, and operational toolin g.Develop dashboards and visualization tools that provide insight into datasets, pipeline health, and system performanc e.Contribute to React-based administrative tools and workflow management features that improve dataset management and search capabilitie s.Partner with ML researchers and cross-functional engineering teams to integrate new embedding technologies and deliver scalable software solution s. About the R oleThe Autonomy Behavior ML Data Optimization team is looking for a Software Engineer with strong data processing and pipeline engineering skills to build, scale, and optimize ScenarioScout -- the scenario discovery platform that empowers teams across the autonomous vehicle development lifecyc le.ScenarioScout transforms large-scale driving data into actionable insights by enabling fast, semantic similarity search over millions of driving scenario embeddin gs.At the heart of this vision, the Autonomy Behavior ML Data Optimization team develops models that forecast the behavior of all road agents around the robotaxi. ScenarioScout is the useful data tool that enables teams to discover, curate, and validate driving scenarios -- from finding diverse training data for ML models to investigating safety-critical edge cases and enriching Focus Area (FA) datase ts.The core of this role is data pipeline engineering: designing and operating Airflow DAGs that orchestrate end-to-end dataset creation and refresh pipelines, building Ray-based distributed processing jobs for k-means clustering, embedding cache generation, FAISS index building, and dataset statistics computation over large Parquet datasets stored in S3.The engineer will also work on the Python/FastAPI backend powering high-throughput embedding search and bulk operations, and contribute to the React.js frontend for UI-driven pipeline management, admin panel features, and search experience improvemen ts.The ideal candidate is someone who thrives in data-intensive environments and is comfortable working across the stack when needed, with particular depth in data processing and pipeline orchestrati on.If you are excited about building data pipelines and processing systems that directly accelerate safe autonomous driving, enjoy working with large-scale distributed data infrastructure, and want to make a measurable impact on how a world-class AV company discovers and understands driving scenarios, this role is for y ou. Responsibili tiesBuild and maintain the Python/FastAPI backend powering high-throughput embedding search, bulk execution APIs, and dataset management endpoi nts.Develop, maintain, and enhance dashboards and data visualization tools for data introspection and ad-hoc report ing.Implement observability and monitoring (pipeline health, search hit rate, result download rate, system health metrics) to ensure platform reliability and data-driven iterat ion.Design, build, and maintain Airflow DAGs that orchestrate end-to-end dataset creation and refresh pipelines, including embedding generation, distributed k-means clustering, FAISS index building, embedding cache construction, and dataset statistics computat ion.Optimize data processing performance and pipeline reliability, including embedding generation pipeline improvements, cache building optimizations, and index construction tun ing.Enhance and operate scalable data processing pipelines for distributed k-means clustering, embedding cache building, FAISS index generation, and dataset statistics computation on large Parquet datasets stored in S3.Own the ScenarioScout dataset lifecycle: full dataset creation, incremental refresh, bulk clustering assignment, dataset registry management, and embedding onboarding for new embedding types (e.g., non-QTP embeddin gs).Collaborate with ML researchers on integrating new embedding types, improving embedding quality, and exploring LLM-powered natural language querying capabilit ies.Contribute to the React.js frontend for admin panel features (dataset creation forms, pipeline monitoring, metadata field configuration), search experience improvements, and UI-driven pipeline managem ent.Deliver UI/UX improvements informed by user feedback, including visualization enhancements, visualization toggles, query result deduplication, shareable search result links, search interruption controls, and result grouping for overlapping scenar ios.Work closely with cross-functional stakeholders (Prediction, Data Optimization, Safety, QA) to understand requirements and translate them into scalable, maintainable softw are. Qualifica tions3+ years of professional software engineering experience with a focus on data processing and pipeline enginee ring.Experience designing, building, and optimizing distributed data processing pipelines at scale (e.g., ETL/ELT) using technologies like Spark, Databricks, AWS EMR, AWS Batch, or Ray Core/ Data.Strong proficiency in Python with experience building production data pipelines and web services (FastAPI, Uvicorn, or similar async framewo rks).Experience building and maintaining data visualization dashboards, with proficiency in SQL, and familiarity with PySpark/Scala for large-scale data manipula tion.Experience with workflow orchestration tools (Airflow, Prefect, Dagster, or similar) for managing complex multi-step data processing pipel ines.Familiarity with vector similarity search and indexing technologies (FAISS, Annoy, ScaNN, Milvus, or simi lar).Experience working with cloud infrastructure (AWS S3, EC2) and container orchestration (Kubernetes, Doc ker).Strong understanding of data structures, algorithms, and performance optimization for data-intensive workl oads.Familiarity with React.js or a comparable modern frontend framework for building interactive data-driven applicat ions.Excellent communication and collaboration skills; ability to work effectively with ML researchers, data engineers, and product stakehol ders.