Staff Research Engineer, Autonomy VLM

Vision\-Language Models (VLMs) are a foundational pillar of our Autonomy stack. In this Staff

Research Engineer role, you will play a key role in delivering the overarching VLM strategy,

especially training, shipping, optimizing the VLM models, as well as extending to

multi\-modalities and enabling new use cases, among others. In this role, you will also be

responsible to define and deliver VLM\-driven solutions to solve some of autonomy's hardest

challenges, including automated data mining, handling long\-tail distributions, rare edge\-case

detection, and scene anomaly reasoning. As part of the model delivery, you will also own the

whole end\-to\-end lifecycle of VLM model delivery: data acquisition, metrics definition,

benchmarking, model performance optimization, deployment, feedback loop.* Drive and deliver the VLM model strategy: Define, drive and execute the roadmap of

VLM model delivery, including training and delivering VLM models, optimization,

deployment, as well as the extension to other multi\-modalities.

• Accelerate data mining: Design and deliver VLM/LLM related models and strategies

that power automated data mining, long\-tail distributions, rare/edge case detection, and

anomaly detection at scale, across multiple modalities (vision, lidar, text, etc).

• Iterate and optimize performance: Establish rigorous evaluation and monitoring

benchmarks. Identify and root\-cause top\-tier system anomalies, prioritizing high\-impact

optimizations to continuously push the needle on performance.

• Cross\-functional collaboration: Partner closely with core Autonomy teams

(Perception, Planning, Calibration, Systems, etc) to translate vehicle feature

requirements into concrete ML deliverables.

• Influence trade\-offs \& requirements: Define system requirements and guide

cross\-functional efforts through technical trade\-off decisions.

Education: BS, MS, or PhD in Computer Science, Robotics, Electrical Engineering, or a

highly related quantitative field.

• Experience: 5\+ years of professional experience scaling ML solutions, with a strong

focus on the following:

• VLM model training: Hands\-on experience training or fine\-tuning VLMs using

modern parameter\-efficient techniques (LoRA, QLoRA) and RL alignment.* Large\-scale data mining: Proven track record developing VLM/LLM\-related

techniques for data mining, long\-tail distributions, rare cases, safety\-critical

events.

• Zero/few\-shot capabilities: Experience with open\-vocabulary, zero\-shot, or

few\-shot classification models, particularly in long\-tail scenarios.

• System engineering: Strong proficiency in Python alongside a solid

understanding of modern Perception pipelines, benchmarking tools, and

infrastructure.

• Execution: Demonstrated ability to root\-cause complex issues across a

distributed, cross\-functional stack in a fast\-paced environment.

Preferred Qualifications

• Experience applying VLMs within the Autonomous Vehicle domain.
• Experience with Auto Prompt Optimization (APO) and automated prompt engineering

techniques.

• Experience with spatial grounding in 2D and/or 3D.
• Experience extending foundational models to extra modalities (e.g., LiDAR, Radar, IMU,

ego\-motion).

• Experience utilizing VLMs or Foundation Models for complex behavior reasoning and

planning.

• Experience with onboard edge deployment, cloud inference architectures, and balancing

compute/efficiency trade\-offs.

• Experience with quantization techniques (PTQ, QAT) and high\-performance inference

engines like TensorRT.