Member of Technical Staff - Infrastructure Reliability
xAI · Palo Alto, CA
About xAI
xAI’s mission is to create AI systems that can accurately understand the universe and aid humanity in its pursuit of knowledge. Our team is small, highly motivated, and focused on engineering excellence. This organization is for individuals who appreciate challenging themselves and thrive on curiosity. We operate with a flat organizational structure. All employees are expected to be hands-on and to contribute directly to the company’s mission. Leadership is given to those who show initiative and consistently deliver excellence. Work ethic and strong prioritization skills are important. All employees are expected to have strong communication skills. They should be able to concisely and accurately share knowledge with their teammates.
ABOUT THE ROLE:
We are training some of the largest models in the world on the latest hardware across multiple environments. To do this reliably at xAI’s pace, we need engineers who have battle-tested experience keeping massive distributed infrastructure up and running 24/7, including on-prem and cloud-based infrastructure. This is a joint xAI/X role: you will own 24×7 reliability for the world’s largest GPU training superclusters and one of the highest-QPS production systems on the planet (X).
You will own the availability, performance, and evolution of xAI’s core compute, storage, and networking infrastructure. This is not an ops-only role — strong coding is a hard requirement. You will design, implement, and ship systems software, automation, and tooling in Python and/or Rust that directly impact training throughput and cluster utilization. You will be expected to participate in a team on-call rotation and to contribute to ushering xAI into the next generation of infrastructure management across multiple data centers and cloud environments.
RESPONSIBILITIES:
- Define and execute the technical strategy for infrastructure reliability and scalability
- Build and maintain the automation, observability, and control planes that keep multi-datacenter, hybrid cloud/on-prem environments healthy
- Lead incident response, deep-dive root cause analysis, and post-mortems that drive real fixes
- Identify, instrument, and eliminate systemic failure patterns (capacity, network, hardware, storage, software)
- Design and implement high-leverage systems software (daemons, controllers, schedulers, etc.) in Python and Rust.
- Push the state of the art in large-scale GPU cluster operations and AI workload reliability
BASIC QUALIFICATIONS:
- 5+ years shipping production software and/or operating distributed infrastructure at scale
- Expert-level knowledge of Linux systems, TCP/IP networking, and systems programming
- Strong coding skills with proven production experience in Rust (strongly preferred) and at least one of Python, Go, or C++.
- Deep experience with large-scale distributed systems in on-prem and cloud environments (GCP experience a plus)
- Hands-on expertise with container orchestration (Kubernetes, Borg-class systems, or custom schedulers), container runtimes, and infrastructure-as-code (Puppet/Chef/Ansible/Terraform)
- Intimate understanding of common failure modes in distributed systems and how to mitigate them (blast radius control, failure domains, canaries, chaos engineering, etc.)
- Track record of participating in (or building) effective on-call rotations in high-stakes environments
- Bachelor’s degree in Computer Science, Electrical Engineering, or equivalent real-world experience
PREFERRED SKILLS AND EXPERIENCE:
- Significant contributions to large-scale GPU clusters or AI/ML infrastructure
- Experience in on-call rotations and incident response in high-stakes environments.
- Strong problem-solving skills and ability to thrive in a fast-paced, ambiguous setting.
- Experience with high-performance networking (RDMA, RoCE, Infiniband) and low level configuration (eBPG, xdp, io_uring)
- Comfortable with deployment, support, monitoring, administration, and troubleshooting across on-prem, cloud and hybrid infrastructures.
COMPENSATION AND BENEFITS:
$180,000 - $400,000 USD
Base salary is just one part of our total rewards package at xAI, which also includes equity, comprehensive medical, vision, and dental coverage, access to a 401(k) retirement plan, short & long-term disability insurance, life insurance, and various other discounts and perks.
xAI is an equal opportunity employer. For details on data processing, view our Recruitment Privacy Notice.
