Kubernetes on Oxide: How Customer Needs Shaped Our Integrations

No single provisioning approach fit all customers' workflows, so we ended up publishing three integrations.

Provisioning integrations create and manage Oxide instances, but they do not reconcile those instances with Kubernetes Node objects. Without this reconciliation, a cluster cannot determine the status of an unreachable Kubernetes node.

We needed a component that ran in each cluster and used the Oxide API to reconcile Oxide infrastructure with Kubernetes state. Kubernetes provides the cloud controller manager (CCM) for exactly this purpose. A CCM integrates Kubernetes resources with an infrastructure provider’s API.

We built the Oxide cloud controller manager to integrate Kubernetes with Oxide. Its node controller keeps Kubernetes Node objects synchronized with their Oxide instances, recording details such as instance IDs, network addresses, and instance state. Kubernetes uses this information to initialize nodes and remove them from the cluster when their Oxide instances are deleted.

The CCM does not create instances or provision clusters. That remains the job of provisioning integrations such as the Rancher node driver, the Omni infrastructure provider, and CAPOx. The CCM works with all the provisioning integrations and gives us an integration point to build upon. As Oxide evolves, we can add new controllers to the CCM rather than update every provisioning integration.

With that runtime extension point in place, we could address another layer of the Kubernetes experience: exposing applications. The CCM architecture also defines a service controller for Kubernetes LoadBalancer services, giving us a place to address the next customer problem.

One of the capabilities customers expect from cloud-integrated Kubernetes is support for Service objects of type LoadBalancer. When a user creates one, Kubernetes asks the CCM’s service controller to provision the load balancer and publish its address in the Service status. There was just one problem: Oxide did not yet offer a native load balancer.

Oxide did, however, have floating IPs. Floating IPs are external addresses that can be attached to and detached from instances, exposing those instances outside the VPCs. I decided to use floating IPs for LoadBalancer services to work around Oxide’s lack of a native load balancer. The idea was that a floating IP would deliver traffic to a single Kubernetes node, and the Kubernetes Service would distribute that traffic to the appropriate pods.

Making that work required accounting for how Oxide floating IPs appear to an instance. They are transparent to the guest in two important ways. First, Oxide translates the destination address of inbound traffic to the instance’s internal IP before sending the traffic to the instance. Second, the instance has no network interface configured with the floating IP.

The resulting traffic flow looks like this:

Since traffic arrives at the instance using its internal IP, the service controller publishes two entries in status.loadBalancer.ingress:^[4]

The status entries look like this:

This architecture also makes the kubectl output look a little unusual:

Users see both the floating IP and the node’s internal IP in the EXTERNAL-IP column, even though only the floating IP is externally reachable. This is an imperfect abstraction, but it allows us to support a common Kubernetes workflow while waiting for a native Oxide load balancer.

This implementation currently supports externalTrafficPolicy: Cluster,^[5] which allows the selected node to forward traffic to a Service endpoint anywhere in the cluster. If that node disappears, the CCM moves the floating IP to another eligible node and updates the internal address in the Service status.

When we introduce a native Oxide load balancer, we can update the service controller to use it without changing the Kubernetes interface. Customers will continue creating the same LoadBalancer services, this time powered by the Oxide load balancer.

To install the Oxide CCM on your cluster, see our CCM guide to get started.

With clusters provisioned, reconciled with Oxide, and reachable from outside their VPCs, storage for stateful workloads became the next problem to address. Kubernetes users request persistent storage through PersistentVolumeClaim objects and expect a Container Storage Interface (CSI) plugin to create, attach, and mount the underlying volumes. Oxide had disks, but Kubernetes had no native way to manage their lifecycle.

Without an Oxide CSI plugin, customers could deploy a third-party Kubernetes storage system such as Longhorn. Longhorn provides its own CSI plugin that uses existing disks and handles its own replication. The only disk Oxide supported at the time was a distributed disk which replicates data across three sleds. Using Longhorn with distributed disks created substantial write fan-out that customers wanted to eliminate.

Oxide local disks gave us a way to reduce write amplification when using Longhorn. Local disks have no built-in replication, making them well suited for Longhorn. Our Rancher showcase uses this architecture to reduce write amplification. While this works today, customers still wanted a native Oxide CSI plugin.

My teammate Luiz wrote RFD 595 Oxide CSI Plugin, describing how an Oxide CSI plugin would work. Everything seemed straightforward on paper, a user creates a PersistentVolumeClaim and the CSI controller creates an Oxide distributed disk, attaches it to the node running the pod, and formats it for use. If the pod is rescheduled to another node, the CSI controller detaches the disk and reattaches it to the new node.

Further discussion and prototyping exposed a blocker. Oxide requires an instance to be stopped before attaching or detaching disks. Kubernetes, however, expects a CSI plugin to attach storage to a running node. Stopping the node would disrupt every other workload and could trigger cascading scheduling and attachment operations.

Before we can release our CSI plugin, we need to add support for disk hot-plug throughout the Oxide stack, from the hypervisor all the way up to the API. What began as a Kubernetes integration has turned into a project spanning multiple layers of the Oxide software stack.

Disk hot-plug and the Oxide CSI plugin remain under active development as of this writing. In the meantime, customers can use software such as Longhorn with Oxide local disks for dynamically provisioned persistent storage without stacking two layers of replication. When the native CSI plugin ships, customers will be able to use familiar Kubernetes storage APIs backed directly by Oxide distributed disks with replication and durability built in.

We now have a growing ecosystem of Kubernetes integrations that work well with one another. Rancher, Omni, and Cluster API solve Kubernetes provisioning, the Oxide CCM reconciles Kubernetes nodes with Oxide instances and handles LoadBalancer services. Customers already use some of these integrations in production, and we dogfood several in our own production workloads. Together, they provide a solid foundation to build on.

Our next step is to expand our dogfooding with the newly released Cluster API provider. Using it to provision and operate more of our clusters will test how these integrations work together day to day.

We still have plenty to build and polish. Our near-term work includes completing disk hot-plug and shipping the CSI plugin, adding autoscaling support, and extending the CCM service controller to support external subnets. Longer term, as we ship resource tagging, OIDC support, and native load balancing, we’ll extend our Kubernetes integrations to take advantage of them.

Building these integrations showed how the architectures of Kubernetes and Oxide complement one another. Kubernetes gives infrastructure providers standard extension points, while Oxide exposes infrastructure primitives through APIs. Oxide’s hardware and software co-design lets us address integration blockers at the layer where they belong and carry the necessary changes through the full stack.

This work also lets us exercise our SDKs and APIs from our customers' perspectives and turn customer friction into product improvements. That feedback loop is how we will continue growing this ecosystem. Customer needs shaped each integration in this post, and they will shape the next one, too.

To see the Cluster API and cloud controller manager integrations in action, watch the video below, in which I deploy a Kubernetes cluster on Oxide.