Software is hard (yes, even in an era of vibe coding), and systems software — the silent engine room of modern infrastructure — is especially so. By design, systems software provides an abstraction for programs, insulating programmers from the filthy details that lie beneath; piercing that abstraction to implement the underlying system is to embrace those details and their gnarly implications. Moreover, the expectation for systems software is (rightly) perfection; a system that is merely functional can be deceptively distant from the robustness required of foundational software. Systems software isn’t the only kind of hard software, of course, and indeed software can be difficult just by nature of its scope and composition: it is hard to build software that is just…​ big. Software that consists of many different modules and components built by multiple people over an extended period of time is known as programming in the large, and its difficulties extend beyond the mere implementation challenges of systems software.

Because the difficulties with developing systems software are broadly orthogonal to those of programming in the large, intersecting these two challenges — that is, developing systems software in the large — is to take on the most grueling of projects: it is the stuff of which mythical man months are literally made. Why would anyone ever develop such a system? Because they are often necessary to tackle software’s equivalents of the wicked problem: problems that are not only never completely solved, but also not even really understood until implementation is well underway. There are not pat answers for developing these systems — nor, infamously, silver bullets — they’re just…​ brutal.

This is on my mind because of a talk that we had at OxCon last week. OxCon is our affectionate name for the annual Oxide meetup here in Emeryville, and it’s a highlight of the year for everyone at Oxide. This year more than lived up to our high expectations, replete with cameos from the extraordinary IBM 26 Interpreting Card Punch and the Oakland Ballers. At OxCon we like to both reflect back and look forward, so in that spirit, we asked Oxide engineer Dave Pacheco if he might be willing to present on the project he’s been leading the charge on for the past two years: software update.

When we shipped the first Oxide rack two years ago, it had the minimum functionality necessary to update all its software in the field. Our priority was to make this update mechanism robust over all else, and we succeeded in the sense that it is indeed robust — but the experience is not yet the seamless, self-service facility that we have envisioned. Software update for the Oxide rack is exactly the kind of wicked problem that necessitates systems software in the large: it is not merely dynamically overhauling a distributed system, but doing so while remaining operable in the liminal state between the old software and the new. Compounding this was the urgency we felt: delivering self-service update is essential to realize our vision of the cloud experience on premises, and our customers needed it as soon as we could deliver it. As if this weren’t enough, the Oxide update problem has an acute constraint not faced by the public clouds: we need to be able to deliver updates across an air gap — we cannot rely on the public cloud’s hidden crutch of operators and runbooks. It is a problem so wicked, you can practically hear it cackle.

Despite the thorniness of the problem, Dave and team had managed to achieve the ambitious milestones that they had set for themselves at OxCon last year, and I was naturally excited for his presentation this year. That said, I wasn’t ready for what was coming: Dave not only described the tremendous work on software update (delving into both the multi-year history of the project and the significant progress since the last OxCon), but also reflected on leading the software update project itself. The result was an absolutely extraordinary talk, not just on the mechanics of software essential to Oxide, but on the unique challenges of systems software in the large.

Dave’s talk dripped with hard-won wisdom, running the gamut from maintaining focus (and the looming specter of what Dave calls "organizational procrastination") to fighting scope creep and the mechanics of specific technical decisions. We felt Dave’s talk to be too good to be kept to ourselves — and thanks to our transparency, nothing in it needs to be secret; we are thrilled to be able to make it generally available:

This talk is a must watch for anyone doing systems software in the large, containing within it the kind of lessons that are often only learned the hard way. While we think it’s valuable for everyone, should you be the kind of sicko inexplicably drawn to exactly the kind of nasty problems that Dave describes, consider joining us — there is more systems software in the large to be done at Oxide!

We don’t want to bury the lede: we have raised a $100M Series B, led by a new strategic partner in USIT with participation from all existing Oxide investors. To put that number in perspective: over the nearly six year lifetime of the company, we have raised $89M; our $100M Series B more than doubles our total capital raised to date — and positions us to make Oxide the generational company that we have always aspired it to be.

If this aspiration seems heady now, it seemed absolutely outlandish when we were first raising venture capital in 2019. Our thesis was that cloud computing was the future of all computing; that running on-premises would remain (or become!) strategically important for many; that the entire stack — hardware and software — needed to be rethought from first principles to serve this market; and that a large, durable, public company could be built by whomever pulled it off.

This scope wasn’t immediately clear to all potential investors, some of whom seemed to latch on to one aspect or another without understanding the whole. Their objections were revealing: "We know you can build this," began more than one venture capitalist (at which we bit our tongue; were we not properly explaining what we intended to build?!), "but we don’t think that there is a market."

Entrepreneurs must become accustomed to rejection, but this flavor was particularly frustrating because it was exactly backwards: we felt that there was in fact substantial technical risk in the enormity of the task we put before ourselves — but we also knew that if we could build it (a huge if!) there was a huge market, desperate for cloud computing on-premises.

Fortunately, in Eclipse Ventures we found investors who saw what we saw: that the most important products come when we co-design hardware and software together, and that the on-premises market was sick of being told that they either don’t exist or that they don’t deserve modernity. These bold investors — like the customers we sought to serve — had been waiting for this company to come along; we raised seed capital, and started building.

And build it we did, making good on our initial technical vision:

While these technological components are each very important (and each is in service to specific customer problems when deploying infrastructure on-premises), the objective is the product, not its parts. The journey to a product was long, but we ticked off the milestones. We got the boards brought up. We got the switch transiting packets. We got the control plane working. We got the rack manufactured. We passed FCC compliance.

And finally, two years ago, we shipped our first system!

Shortly thereafter, more milestones of the variety you can only get after shipping: our first update of the software in the field; our first update-delivered performance improvements; our first customer-requested features added as part of an update.

Later that year, we hit general commercial availability, and things started accelerating. We had more customers — and our first multi-rack customer. We had customers go on the record about why they had selected Oxide — and customers describing the wins that they had seen deploying Oxide.

Customers started landing faster now: enterprise sales cycles are infamously long, but we were finding that we were going from first conversations to a delivered product surprisingly quickly. The quickening pace always seemed to be due in some way to our transparency: new customers were listeners to our podcast, or they had read our RFDs, or they had perused our documentation, or they had looked at the source code itself.

With growing customer enthusiasm, we were increasingly getting questions about what it would look like to buy a large number of Oxide racks. Could we manufacture them? Could we support them? Could we make them easy to operate together?

Into this excitement, a new potential investor, USIT, got to know us. They asked terrific questions, and we found a shared disposition towards building lasting value and doing it the right way. We learned more about them, too, and especially USIT’s founder, Thomas Tull. The more we each learned about the other, the more there was to like. And importantly, USIT had the vision for us that we had for ourselves: that there was a big, important market here — and that it was uniquely served by Oxide.

We are elated to announce this new, exciting phase of the company. It’s not necessarily in our nature to celebrate fundraising, but this is a big milestone, because it will allow us to address our customers' most pressing questions around scale (manufacturing scale, system scale, operations scale) and roadmap scope. We have always believed in our mission, but this raise gives us a new sense of confidence when we say it: we’re going to kick butt, have fun, not cheat (of course!), love our customers — and change computing forever.

We are heartbroken to relay that Charles Beeler, a friend and early investor in Oxide, passed away in September after a battle with cancer. We lost Charles far too soon; he had a tremendous influence on the careers of us both.

Our relationship with Charles dates back nearly two decades, to his involvement with the ACM Queue board where he met Bryan. It was unprecedented to have a venture capitalist serve in this capacity with ACM, and Charles brought an entirely different perspective on the practitioner content. A computer science pioneer who also served on the board took Bryan aside at one point: "Charles is one of the good ones, you know."

When Bryan joined Joyent a few years later, Charles also got to know Steve well. Seeing the promise in both node.js and cloud computing, Charles became an investor in the company. When companies hit challenging times, some investors will hide — but Charles was the kind of investor to figure out how to fix what was broken. When Joyent needed a change in executive leadership, it was Charles who not only had the tough conversations, but led the search for the leader the company needed, ultimately positioning the company for success.

Aside from his investment in Joyent, Charles was an outspoken proponent of node.js, becoming an organizer of the Node Summit conference. In 2017, he asked Bryan to deliver the conference’s keynote, but by then, the relationship between Joyent and node.js had become…​ complicated, and Bryan felt that it probably wouldn’t be a good idea. Any rational person would have dropped it, but Charles persisted, with characteristic zeal: if the Joyent relationship with node.js had become strained, so much more the reason to speak candidly about it! Charles prevailed, and the resulting talk, Platform as Reflection of Values, became one of Bryan’s most personally meaningful talks.

Charles’s persistence was emblematic: he worked behind the scenes to encourage people to do their best work, always with an enthusiasm for the innovators and the creators. As we were contemplating Oxide, we told Charles what we wanted to do long before we had a company. Charles laughed with delight: "I hoped that you two would do something big, and I am just so happy for you that you’re doing something so ambitious!"

As we raised seed capital, we knew that we were likely a poor fit for Charles and his fund. But we also knew that we deeply appreciated his wisdom and enthusiasm; we couldn’t resist pitching him on Oxide. Charles approached the investment in Oxide as he did with so many other aspects: with curiosity, diligence, empathy, and candor. He was direct with us that despite his enthusiasm for us personally, Oxide would be a challenging investment for his firm. But he also worked with us to address specific objections, and ultimately he won over his partnership. We were thrilled when he not only invested, but pulled together a syndicate of like-minded technologists and entrepreneurs to join him.

Ever since, he has been a huge Oxide fan. Befitting his enthusiasm, one of his final posts expressed his enthusiasm and pride in what the Oxide team has built.

Charles, thank you. You told us you were proud of us — and it meant the world. We are gutted to no longer have you with us; your influence lives on not just in Oxide, but also in the many people that you have inspired. You were the best of venture capital. Closer to the heart, you were a terrific friend to us both; thank you.

Paul Graham’s Founder Mode is an important piece, and you should read it if for no other reason that "founder mode" will surely enter the lexicon (and as Graham grimly predicts: "as soon as the concept of founder mode becomes established, people will start misusing it"). When building a company, founders are engaged in several different acts at once: raising capital; building a product; connecting that product to a market; building an organization to do all of these. Founders make lots of mistakes in all of these activities, and Graham’s essay highlights a particular kind of mistake in which founders are overly deferential to expertise or convention. Pejoratively referring to this as "Management Mode", Graham frames this in the Silicon Valley dramaturgical dyad of Steve Jobs and John Scully. While that’s a little too reductive (anyone seeking to understand Jobs needs to read Randall Stross’s superlative Steve Jobs and the NeXT Big Thing, highlighting Jobs’s many post-Scully failures at NeXT), Graham has identified a real issue here, albeit without much specificity.

For a treatment of the same themes but with much more supporting detail, one should read the (decade-old) piece from Tim O’Reilly, How I failed. (Speaking personally, O’Reilly’s piece had a profound influence on me, as it encouraged me to stand my ground on an issue on which I had my own beliefs but was being told to defer to convention.) But as terrific as it is, O’Reilly’s piece also doesn’t answer the question that Graham poses: how do founders prevent their companies from losing their way?

Graham says that founder mode is a complete mystery ("There are as far as I know no books specifically about founder mode"), and while there is a danger in being too pat or prescriptive, there does seem to be a clear component for keeping companies true to themselves: the written word. That is, a writing- (and reading-!) intensive company culture does, in fact, allow for scaling the kind of responsibility that Graham thinks of as founder mode. At Oxide, our writing-intensive culture has been absolutely essential: our RFD process is the backbone of Oxide, and has given us the structure to formalize, share, and refine our thinking. First among this formalized thinking – and captured in our first real RFD – is RFD 2 Mission, Principles, and Values. Immediately behind that (and frankly, the most important process for any company) is RFD 3 Oxide Hiring Process. These first three RFDs – on the process itself, on what we value, and on how we hire – were written in the earliest days of the company, and they have proven essential to scale the company: they are the foundation upon which we attract people who share our values.

While the shared values have proven necessary, they haven’t been sufficient to eliminate the kind of quandaries that Graham and O’Reilly describe. For example, there have been some who have told us that we can’t possibly hire non-engineering roles using our hiring process – or told us that our approach to compensation can’t possibly work. To the degree that we have had a need for Graham’s founder mode, it has been in those moments: to stay true to the course we have set for the company. But because we have written down so much, there is less occasion for this than one might think. And when it does occur – when there is a need for further elucidation or clarification – the artifact is not infrequently a new RFD that formalizes our newly extended thinking. (RFD 68 is an early public and concrete example of this; RFD 508 is a much more recent one that garnered some attention.)

Most importantly, because we have used our values as a clear lens for hiring, we are able to assure that everyone at Oxide is able to have the same disposition with respect to responsibility – and this (coupled with the transparency that the written word allows) permits us to trust one another. As I elucidated in Things I Learned The Hard Way, the most important quality in a leader is to bind a team with mutual trust: with it, all things are possible – and without it, even easy things can be debilitatingly difficult. Graham mentions trust, but he doesn’t give it its due. Too often, founders focus on the immediacy of a current challenge without realizing that they are, in fact, undermining trust with their approach. Bluntly, founders are at grave risk of misinterpreting Graham’s "Founders Mode" to be a license to micromanage their teams, descending into the kind of manic seagull management that inhibits a team rather than empowering it.

Founders seeking to internalize Graham’s advice should recast it by asking themselves how they can foster mutual trust – and how they can build the systems that allow trust to be strengthened even as the team expands. For us at Oxide, writing is the foundation upon which we build that trust. Others may land on different mechanisms, but the goal of founders should be the same: build the trust that allows a team to kick a Jobsian dent in the universe!

We ran into an interesting issue recently. On the one hand, it was routine: we had a bug — a regression — and the team quickly jumped on it, getting it root caused and fixed. But on the other, this particular issue was something of an Oxide object lesson, representative not just of the technologies but also of the culture we have built here. I wasn’t the only person who thought so, and two of my colleagues wrote terrific blog entries with their perspectives:

The initial work as described by Matt represents a creative solution to a thorny problem; if it’s clear in hindsight, it certainly wasn’t at the time! (In Matt’s evocative words: "One morning, I had a revelation.") I first learned of Matt’s work when he demonstrated it during our weekly Demo Friday, an hour-long unstructured session to demo our work for one another. Demo Friday is such an essential part of Oxide’s culture that it feels like we have always done it, but in fact it took us nearly two years into the company’s life to get there: over the spring and summer of 2021, our colleague Sean Klein had instituted regular demos for the area that he works on (the Oxide control plane), and others around the company — seeing the energy that came from it — asked if they, too, could start regular demos for their domain. But instead of doing it group by group, we instituted it company-wide starting in the fall of 2021: an unstructured hour once a week in which anyone can demo anything.

In the years since, we have had demos of all scopes and sizes. Importantly, no demo is too small — and we have often found that a demo that feels small to someone in the thick of work will feel extraordinary to someone outside of it. ("I have a small demo building on the work of a lot of other people" has been heard so frequently that it has become something of an inside joke.) Demo Friday is important because it gets to one of our most important drivers as technologists: the esteem of our peers. The thrill that you get from showing work to your colleagues is unparalleled — and their wonderment in return is uniquely inspiring. (Speaking personally, Matt’s demo addressed a problem that I had personally had many times over in working on Hubris — and I was one of the many w00ts in the chat, excited to see his creative solution!)

Having the demos be company-wide has also been a huge win for not just our shared empathy and teamwork but also our curiosity and versatility: it’s really inspiring to have (say) one colleague show how they used PCB backdrilling for signal integrity, and the next show an integration they built using Dropshot between our CRM and spinning up a demonstration environment for a customer. And this is more than just idle intellectual curiosity: our stack is deep — spanning both hardware and software — and the demos make for a fun and engaging way to learn about aspects of the system that we don’t normally work on.

Returning to Matt and Cliff, if Matt’s work implicitly hits on aspects of our culture, Cliff’s story of debugging addresses that culture explicitly, noting that the experience demonstrated:

In the discussion on Hacker News of Cliff’s piece, this cultural observeration stood out, with a commenter asking:

The culture at Oxide is in fact very deliberate: when starting a company, one is building many things at once (the team, the product, the organization, the brand) — and the culture will both inform and be reinforced by all of these. Setting that first cultural cornerstone was very important to us — starting with our mission, principles, and values. Critically, by using our mission, principles, and values as the foundation for our hiring process, we have deliberately created a culture that reinforces itself.

Some of the implementation details:

If it needs to be said, there is plenty of ambiguity: if you are using absolutes to think of Oxide (outside of our principles of honesty, integrity and decency!) you are probably missing some nuance of our culture.

Finally, to the (seemingly loaded?) question of the "drawbacks" of fostering "openness, curiosity, and communication" within an organization, the only drawback is that it’s hard work: culture has to be deliberate without being overly prescriptive, and that can be a tricky balance. In this regard, building a culture is very different than building (say) software: it is not engineered in a traditional sense, but is rather a gooey, squishy, organism that will evolve over time. But the reward of the effort is something that its participants care intensely about: it will continue to be (in Cliff’s words) a culture that we work hard to not just create but defend!

Years ago, Jeff Bezos famously quipped that "your margin is my opportunity." This was of course aimed not at Amazon’s customers, but rather its competitors, and it was deadly serious: customers of AWS in those bygone years will fondly remember that every re:Invent brought with it another round of price cuts. This era did not merely reflect Bezos’s relentless execution, but also a disposition towards who should reap the reward of advances in underlying technology: Amazon believed (if implicitly) that improvements at the foundations of computing (e.g., in transistor density, core count, DRAM density, storage density, etc.) should reflect themselves in lower prices for consumers rather than higher margins for suppliers.

Price cuts are no longer a re:Invent staple, having been replaced by a regular Amazon tradition of a different flavor: cutting depreciation (and therefore increasing earnings) by extending the effective life of their servers. (These announcements are understandably much more subdued, as "my depreciation is my margin opportunity" doesn’t have quite the same ring to it.)

As compute needs have grown and price cuts have become an increasingly distant memory, some have questioned their sky-high cloud bills, wondering if they should in fact be owning their compute instead of renting it. When we started Oxide, we knew from operating our own public cloud what those economics looked like — and we knew that over time others of a particular scale would come to the same realization that they would be better off not giving their margin away by renting compute. (Though it’s safe to say that we did not predict that it would be DHH leading the charge!)

Owning one’s own cloud sounds great, but there is a bit that’s unsaid: what about the software? Software is essential for elastic, automated infrastructure: hardware alone does not a cloud make! Unfortunately, the traditional server vendors do not help here: because of a PC-era divide in how systems are delivered, customers are told to look elsewhere for any and all system software. This divide is problematic on several levels. First, it impedes the hardware/software co-design that we (and, famously, others!) believe is essential to deliver the best possible product. Second, it leads to infamous finger pointing when the whole thing doesn’t work. But there is also a thorny economic problem: when your hardware and your software don’t come from the same provider, to whom should go the spoils of better hardware?

To someone who has just decided to buy their hardware out of their frustration with renting it, the answer feels obvious: whoever owns the hardware should naturally benefit from its advances! Unfortunately, the enterprise software vendor delivering your infrastructure often has other ideas — and because their software is neither rented nor bought, but rather comes from the hinterlands of software licensing, they have broad latitude as to how it is priced and used. In particular, this allows them to charge based on the hardware that you run it on — to have per-core software licensing.

This galling practice isn’t new (and is in fact as old as symmetric multiprocessing systems), but it has taken on new dimensions in the era of chiplets and packaging innovation: the advances that your next CPU has over your current one are very likely to be expressed in core count. Per-core licensing allows a third party — who neither made the significant investment in developing the next generation of microprocessor nor paid for the part themselves — to exact a tax on improved infrastructure. (And this tax can be shockingly brazen!) Couple this with the elimination of perpetual licensing, and software costs can potentially absorb the entire gain from a next-generation CPU, leaving a disincentive to run newer, more efficient infrastructure. As an industry, we have come to accept this practice, but we shouldn’t: in the go-go era of Dennard scaling (when clock rates rose at a blistering rate), software vendors never would have been allowed to get away with charging by the gigahertz; we should not allow them to feel so emboldened to charge by core count now!

If it needs to be said, we have taken a different approach at Oxide: when you buy the Oxide cloud computer, all of the software to run it is included. This includes all of the software necessary to run the rack as elastic infrastructure: virtual compute, virtual storage, virtual networking. (And yes, it’s all open source — which unfortunately demands the immediate clarification that it’s actually open source rather than pretend open source.) When we add a new feature to our software, there is no licensing enablement or other such nuisance — the feature just comes with the next update. And what happens when AMD releases a new CPU with twice the core count? The new sled running the new CPU runs along your existing rack — you’re not paying more than the cost of the new sled itself. This gives the dividends of Moore’s Law (or Wright’s Law!) to whom they rightfully belong: the users of compute.

“We offer the best health insurance we could find” is what we promise in our job postings. On paper, this is accurate: the health insurance Oxide offers is the best plan we can find that is offered to small businesses.

What we left unsaid until now is that the best health insurance offered to small businesses is, in fact, not very good at all if you don’t neatly fit into a narrow demographic; the bitter irony is that the US healthcare system isn’t designed for those of us who rely on it the most. And life-saving treatments that aren’t needed by able-bodied cisgender people are, more often than not, deemed “not medically necessary” in off-the-shelf insurance policies simply because there is no law to require their coverage. In our society’s bizarre and accidental system, we rely on employers to provide benefits everyone should have, including healthcare, retirement plans, and dependent care.

When I came out as trans and started seeking medical care, I worked at a large employer that directly paid for the medical costs of its employees and dictated how their insurance networks process claims. I had the benefits I needed because other trans people fought for them and the company could unilaterally choose to provide them. Startups don’t have this luxury and are at the whims of insurance companies to keep the cost of hiring and retaining employees manageable. Meanwhile, insurers put profit over care, and won’t budge on off-the-shelf benefits plans unless Congress forces the issue – yet as I write this, lawmakers across the country are either ignoring us or actively stripping away our right to get the healthcare we need, so we’re not holding our breath.

With this in mind, I was initially uneasy about applying to Oxide because we didn’t make our benefits clear to prospective applicants. But I still applied — not out of blind faith, but because I felt a company built on these values would put its people first and work to provide what I needed. Shortly after I started, despite insurance companies not budging even an inch, our CEO Steve announced a reimbursement arrangement that would cover $10,000 of out-of-pocket healthcare expenses for gender-related healthcare per year. This isn’t perfect, but it’s a damn good start.

Future applicants shouldn’t need to put this much faith in us upholding our values in order to be comfortable about applying, though. Here’s a much clearer summary of our benefits:

The bottom line: Where our insurers fall short, we will work to meet our employees where they need us to be.

As with our compensation model, the benefits we provide embody our mission, principles, and values: We can’t focus on our mission if we’re distracted by healthcare expenses. Our principles of integrity, honesty, and decency compel us to care for our teammates and their families. And our approach to benefits intersects with several of our values:

Finally, in the spirit of transparency: we’ve made our benefits information public starting today. These are close to the same documents employees see when signing up for benefits (with some information only relevant for employees removed). We’re working to provide additional information for specific situations that we’re aware of — and we know that our benefits are not complete; there are innumerable needs none of us have experienced or thought of.

We share our benefits information for two important reasons: first, we want applicants to be able to learn everything we know about our benefits so that they feel confident in applying even in the face of a healthcare need that is not commonly covered; second, we want to give employees at other similarly-situated companies tools (such as suggesting HRAs) to help fight for the healthcare coverage they need. We have a responsibility to take care of our employees, but we also have a responsibility to make our industry a better place to work for everyone.

When we started Oxide, we knew we were going to take a fresh look at the entire system. We knew, for example, that we wanted to have a true hardware root of trust and that we wanted to revisit the traditional BMC. We knew, too, that we would have our own system software on each of these embedded systems, and assumed that we would use an existing open source operating system as a foundation. However, as we waded deeper into these embedded systems and especially their constraints of security and robustness we found that what we wanted out of the existing operating systems wasn’t necessarily what they offered (or even wanted to offer).

As time went on in early 2020 and we found ourselves increasingly forcing existing systems out of the comfort of their design centers, we wondered: was our assumption of using an existing system wrong? Should we in fact be exploring our own de novo operating system? In particular, our colleague Cliff Biffle, who had a ton of experience with both Rust and embedded systems, had a vision for what such a system might look like (namely, the system that he had always wanted for himself!). Cliff dove into a sketch of his ideas, giving the nascent system a name that felt perfectly apt: Hubris.

After just a few weeks, Cliff’s ideas were taking concrete shape, and it was clear that there was a lot to like: the emerging Hubris was an all-Rust system that was not distracting itself by accommodating other runtimes; it was microkernel-based, allowing for safety and isolation; it employed a strictly synchronous task model, allowing for it be easily developed and comprehended; and it was small and light, allowing it to fit into some of the tight spots we envisioned for it. But for me, it was the way Cliff thought about the building of the system that really set Hubris apart: instead of having an operating system that knows how to dynamically create tasks at run-time (itself a hallmark of multiprogrammed, general purpose systems), Cliff had designed Hubris to fully specify the tasks for a particular application at build time, with the build system then combining the kernel with the selected tasks to yield a single (attestable!) image.

This is the best of both worlds: it is at once dynamic and general purpose with respect to what the system can run, but also entirely static in terms of the binary payload of a particular application — and broadly static in terms of its execution. Dynamic resource exhaustion is the root of many problems in embedded systems; having the system know a priori all of the tasks that it will ever see liberates it from not just a major source of dynamic allocation, but also from the concomitant failure modes. For example, in Hubris, tasks can always be safely restarted, because we know that the resources associated with a task are available if that task itself has faulted! And this eliminates failure modes in which dynamic task creation in response to load induces resource exhaustion; as Cliff has quipped, it is hard to have a fork bomb when the system lacks fork itself!

Precedence for the Hubris approach can be found in other systems like library operating systems, but there is an essential difference: Hubris is a memory-protected system, with tasks, the kernel, and drivers all in disjoint protection domains. (And yes, even in memory safe languages like Rust, memory protection is essential!) In this regard, Hubris represents what we like about Rust, too: a creative solution that cuts through a false dichotomy to yield a system that is at once nimble and rigorous.

It was clear that Cliff was on the right track with Hubris, and the rest of us jumped in with gusto. For my own part, with debugging and debuggability deep in my marrow, I set to work writing the debugger that I felt that we would need — and that Hubris deserved. Following Cliff’s lead, I dubbed it Humility, and it’s been exciting for me to see the debugger and the operating system work together to yield a higher-quality system.

We have known for quite some time that we would open source Hubris: not only is open source core to our own commercial thesis at Oxide, we also believe that the open source revolution — and its many advantages for customers — are long overdue in the lowest layers of the software stack. So we were waiting for the right occasion, and the Open Source Firmware Conference afforded us an excellent one: if you are a listener of our On the Metal podcast, you heard us talk about OSFC a bunch, and it felt entirely fitting that we would kickoff our own open source firmware contribution there. And while the conference starts today, the good news is that you haven’t missed anything! Or at least, not yet: the conference is virtual, so if you want to hear Cliff talk about Hubris in his own words — and it’s before 12:10 Pacific today — it’s not too late to buy a ticket! (The recording will naturally be released after the conference.)

And of course, if you just want to play with the system itself, Hubris and Humility are both now open source! Start by checking out the Hubris repo, the Hubris docs, and the Humility repo. (And if you are looking for a hardware vehicle for your exploration, take a look at the ST Nucleo-H753ZI evaluation board — a pretty capable computer for less than thirty bucks!) We believe Rust to be a part of the coming firmware revolution, and it’s exciting for us to have a system out there that embodies that belief!

Compensation: the word alone is enough to trigger a fight-or-flight reaction in many. But we in technology have the good fortune of being in a well-compensated domain, so why does this issue induce such anxiety when our basic needs are clearly covered? If it needs to be said, it’s because compensation isn’t merely about the currency we redeem in exchange for our labors, but rather it is a proxy for how we are valued in a larger organization. This, in turn, brings us to our largest possible questions for ourselves, around things like meaning and self-worth.

So when we started Oxide – as in any new endeavor – compensation was an issue we had to deal with directly. First, there was the thorny issue of how we founders would compensate ourselves. Then, of course, came the team we wished to hire: hybrid local and remote, largely experienced to start (on account of Oxide’s outrageously ambitious mission), and coming from a diverse set of backgrounds and experiences. How would we pay people in different geographies? How could we responsibly recruit experienced folks, many of whom have families and other financial obligations that can’t be addressed with stock options? How could we avoid bringing people’s compensation history – often a reflection of race, gender, class, and other factors rather than capability – with them?

We decided to do something outlandishly simple: take the salary that Steve, Jess, and I were going to pay ourselves, and pay that to everyone. The three of us live in the San Francisco Bay Area, and Steve and I each have three kids; we knew that the dollar figure that would allow us to live without financial distress – which we put at $175,000 a year – would be at least universally adequate for the team we wanted to build. And we mean everyone literally: as of this writing we have 23 employees, and that’s what we all make.

Now, because compensation is the hottest of all hot buttons, it can be fairly expected that many people will have a reaction to this. Assuming you’ve made it to this sentence it means you are not already lighting us up in your local comments section (thank you!), and I want to promise in return that we know some likely objections, and we’ll address those. But before we do, we want to talk about the benefits of transparent uniform compensation, because they are, in a word, profound.

Broadly, our compensation model embodies our mission, principles, and values. First and foremost, we believe that our compensation model reflects our principles of honesty, integrity, and decency. To flip it around: sadly, we have seen extant comp structures in the industry become breeding grounds for dishonesty, deceit, and indecency. Beyond our principles, our comp model is a tangible expression of several of our values in particular:

These are (some of!) the overwhelming positives; what about those objections?

Of these objections, several are of the ilk that this cannot endure at arbitrary scale. This may be true – our compensation may well not be uniform in perpetuity – but we believe wholeheartedly that our values will endure. So if and when the uniformity of our compensation needs to change, we fully expect that it will remain transparent – and that we as a team will discuss it candidly and empathetically. In this regard, we take inspiration from companies that have pioneered transparent compensation. It is very interesting to, for example, look at how Buffer’s compensation has changed over the years. Their approach is different from ours in the specifics, but they are a kindred spirit with respect to underlying values – and their success with transparent compensation gives us confidence that, whatever changes must come with time, we will be able to accommodate them without sacrificing what is important to us!

Finally, a modest correction. The $175,000 isn’t quite true – or at least not anymore. I had forgotten that when we did our initial planning, we had budgeted modest comp increases after the first year, so it turns out, we all got a raise to $180,250 in December! I didn’t know it was coming (and nor did anyone else); Steve just announced it in the All Hands: no three-hundred-and-sixty degree reviews, no stack ranking, no OKRs, no skip-levels, no numerical grades – just a few more organic raspberries in everyone’s shopping basket. Never has a change in compensation felt so universally positive!

UPDATES: Since originally writing this blog entry in 2021, we have increased our salary a few times, and it now stands at $250,000. We have also added some sales positions that have variable compensation, consisting of a lower base salary and a commission component.