At Rhoda AI, we’re building the next generation of generalist intelligent robots. We own the full robotics stack from high-performance hardware and robot systems to the infrastructure and state-of-the-art foundation world models that control our robots. Our robots are designed to be generalists capable of operating in complex, real-world environments and handling long-tail edge cases, made possible by our cutting edge research and end-to-end system design. We've raised over $450M and are investing aggressively in model research, infrastructure, hardware development, and manufacturing scale-up to make generalist robotics a reality.
We're looking for a Staff / Principal ML Training Systems Engineer to own training systems performance end-to-end. You will define how our models train at scale — driving efficiency, scalability, and correctness across large-scale multimodal training. This is a core systems role, not infrastructure support. Your work directly determines how efficiently we use compute, how well models scale across thousands of GPUs, and how quickly research can iterate.
What You'll Do
Own training performance end-to-end
- Diagnose and improve performance of large-scale multimodal training (vision, video, proprioception, actions, language)
- Build systematic performance attribution: step-time decomposition (compute vs communication vs input pipeline), scaling curves across cluster sizes, and bottleneck identification and prioritization
- Drive measurable gains in:Distributed efficiency (comm/compute overlap, bucketization, topology-aware mapping, parallelism strategies)Compute efficiency (kernel hotspots, operator fusion, attention optimization, framework/runtime overhead)Memory efficiency (activation checkpointing, sequence packing/bucketing, fragmentation reduction)
Design training systems (not just tune them)
- Define and evolve parallelism strategies: data / tensor / pipeline / sharding / hybrid approaches
- Improve execution efficiency through communication scheduling and overlap, graph capture and execution optimization, and runtime-level improvements
- Contribute to and extend training frameworks where needed
Make performance observable and measurable
- Establish source-of-truth performance metrics: step-time breakdowns, MFU / throughput / scaling efficiency
- Build tools to identify bottlenecks quickly, track performance across model families, and compare scaling behavior across configurations
