Are you a curious, detail-oriented person who loves shipping polished products? We are seeking a Product Assurance Engineer to work on a multi-disciplinary team to assure that Oxide ships the highest quality product to our customers.
You will contribute to the practice of reliability-oriented testing: designing experiments that surface latent bugs, availability gaps, and performance cliffs before customers encounter them.
As a Product Assurance Engineer working at Oxide, you will:
Design and execute workload-driven stress tests, with a focus on workloads that replicate realistic production traffic patterns such as high connection-rate workloads, micro-burst traffic, large-flow saturation, multicast, and adversarial packet patterns.
Instrument, observe, and correlate metrics that help characterize and pinpoint system issues.
Drive chaos-monkey-style tests to simulate common hardware failure modes and control plane outages.
Investigate availability and reliability gaps, and produce reproducible failure reports.
Work with engineers across the software stack to triage findings and verify fixes and mitigations.
Improve the assurance infrastructure itself: scripts, harnesses, chaos tooling, and automated regression suites.
You will thrive in this role if you:
Have deep familiarity with troubleshooting traffic issues at different network layers and characterising performance and reliability of network data paths.
Are comfortable with low-level debugging, reading packet captures and reasoning from raw protocol state.
Enjoy hands-on work with test infrastructure.
Have a track record of finding non-obvious reliability bugs in complex systems.
Are proficient in at least one scripting or systems language (Python, Rust, Go) for building test harnesses and automating workload generation.
Enjoy writing: clear bug reports, knowledge articles, and technical documentation.
Before applying for this role, you should:
Learn enough about our product to understand our approach.
Listen to Hiring Processes with Gergely Orosz to familiarize yourself with the Oxide hiring process.
Browse our public Requests for Discussion to get a flavor for how we work.
Listen to some of our episodes of Oxide and Friends to get a sense for how we operate. A few recommendations:
Rack-scale Networking to understand our approach to networking and some of the details of our implementation
Hell is other networks to learn about our disposition to customer problems
The Network Behind the Network on the specific challenges of our service processor network
The Tale of Reverso on a particularly gnarly networking bug unearthed by a manufacturing test fixture
Systems Software in the Large on a multi-year effort to develop full-system update
When Nine Nines Isn’t Enough on a multi-year search for an evasive failure
RFDs: The Backbone of Oxide on how we got to our writing-intensive culture