When Cluster Upgrades Stop Moving Together

Every platform release has two audiences, and they read the same notes differently. The first is sizing a new cluster and reads the feature list. The second is already running clusters and reads the prerequisites, because that is where a release decides how the next quarter goes.

NKP 2.19 was announced on October 20, 2026, in the product update on next.nutanix.com, and for the second audience it is a release about the operating contract: each workload cluster now upgrades on its own schedule, the preconditions for getting there are stricter than on 2.18, and one component that used to be opt-in is now on by default.

I read 2.18 the same way in July, as a release that tightened the contract more than it added features; 2.19 raises the stakes on both sides of that contract. For a service provider running clusters for several tenants on one management cluster, that second reading is the one that matters.

What Shipped in October

Stripped of launch copy, 2.19 adds a few things and changes two. NKP Metal provisions Kubernetes on physical servers: hosts are imaged and registered through Foundation Central, placed in an NKP host inventory, assigned to workspaces, and consumed either as full bare-metal clusters or as physical worker pools inside VM-based clusters. Per-cluster upgrades coordinate platform applications, Kubernetes and cluster infrastructure in one workflow, from the UI or a single CLI command, for ClusterClass clusters on AHV and for EKS. Kubernetes 1.36.2 becomes the default, with the 1.35.x family still supported, and Rocky Linux 9.8 and RHEL 9.8 join the node images.

The release also carries a curated AI catalog for Pro and Ultimate (KAI Scheduler, Kueue, Slurm Operator, Kubeflow Pipelines, JupyterHub, Milvus Operator) and GPU scheduling through Dynamic Resource Allocation, without NVIDIA MIG partitioning; those deserve an article with hardware under it rather than release notes.

The two changes are a default and a dashboard. The MCP server shipped in 2.18 as a component of AI Navigator, so it is not a 2.19 feature. What changes today is that on connected Pro and Ultimate management clusters it is enabled out of the box, with a wider read-only reach into Kubernetes, Flux and NKP resources, filtered by RBAC. Kubernetes Dashboard gives way to Headlamp as the platform-managed UI, by default on new deployments. Defaults are what operators inherit without choosing them, which is why both belong in the prerequisites section rather than the feature list.

Per Cluster Is the Operative Phrase

The simplified upgrade is the headline that earns it, and it changes two things: the upgrade takes one command, and it happens per cluster. On 2.18 the NKPCluster model and the preflight checks were already there, but an upgrade still meant several sequenced steps across platform apps, Kubernetes and add-ons, and the estate tended to move together. On 2.19 the documentation states the new contract in one sentence: clusters in the same workspace can run different supported NKP versions and follow different upgrade schedules. The workflow upgrades platform applications first, then Kubernetes and the cluster infrastructure (VMs, ClusterClass resources, control plane, worker pools, CNI, CSI), and worker pools can upgrade in parallel through an annotation on the cluster object.

The ordering constraint stays visible, or rather it becomes the whole structure. The management cluster goes first, and for the 2.18 to 2.19 hop both the Kommander and the management cluster steps are CLI only. The UI path opens for workload clusters only once the management plane is on 2.19. In a provider setup where the management cluster is shared and the workload clusters belong to tenants, the management window is negotiated once and the tenant windows are negotiated separately, which is the shape every MSP wants.

One known issue follows directly from the new freedom: after Kommander is upgraded, the UI and nkp get appdeployment report the newest platform application version for every workspace, including clusters that still run the previous one. The inventory of who is on what has to be read on each cluster, not from either view, until that is fixed.

For pre-provisioned clusters, the ones my toolkit manages, the new piece is nkp upgrade platform-apps followed by the existing Kubernetes and infrastructure steps; that is the upgrade path the toolkit had been waiting for, and it goes into the next release.

The Project Boundary Becomes an Upgrade Gate

The prerequisite with the longest reach concerns Prism Central Projects 2.0. On 2.19 a cluster belongs to exactly one project, either the default project (_internal) or one user-defined project, and that applies to the control plane and every worker pool, including pools added later. A 2.18 cluster whose nodes span projects has to be converged before the upgrade, and the documentation names the typical case: control plane in the default project, GPU pools in a user-defined one.

Two details raise the stakes. Once a project is selected, changing it is not supported; a different project means a new cluster. And a cluster converged into a user-defined project may need Nutanix Support to migrate existing volume groups out of the default project.

I argued in the series anchor, Why the Tenant Exists Before the Control Plane, that the project is the Day 0 decision on pc.7.6, the one that cannot be revisited cheaply afterward. 2.19 turns that argument into an upgrade prerequisite.

An MSP that mapped one tenant to one project and kept every cluster inside its tenant’s project already meets the prerequisite. One that let clusters straddle projects now has a convergence task ahead of the upgrade, and the supported ways to change a node pool’s project are the UI or the cluster object, not the nkp update nodepool command. The constraint is also the guarantee: a cluster that lives entirely inside one project is a cluster whose quota, RBAC and network scope can be reasoned about from one place.

Catalogs Move to OCI

The second structural change is in the application catalog. GitRepository catalogs are no longer supported by the controller or the CLI, and AppDeployment objects have to point at OCIRepository catalogs before the upgrade. For a single platform team this is a Flux source swap. For a provider that publishes a curated catalog into tenant workspaces, it is a change to the publishing pipeline: the catalog becomes an artifact that is built, versioned and pushed to a registry, the same discipline already applied to images.

Two new substitution variables, ${clusterType} and ${clusterUUID}, help the same pipeline template per-cluster values without per-cluster manifests. The removal of the --kommander-applications-repository flag and the deprecation of nkp upgrade workspace point the same way: fewer supported upgrade routes, one supported catalog format. I would plan the catalog migration as its own change in its own window, with the upgrade scheduled after at least one reconcile has gone through the new source.

Smaller Items on the Same Checklist

A few items are small individually and expensive when discovered mid-upgrade. A connected cluster without a registry mirror now pulls platform images from the Nutanix Docker Hub registry with an access token generated on the Support Portal; a management cluster that does not have one has to get it as part of the upgrade, and without it the Kommander upgrade fails. Workload clusters use the management cluster’s token by default and can be given their own, but an organization can hold at most two tokens. So on a shared management cluster the token is a platform credential with a rotation owner: two tokens do not stretch to one per tenant. Air-gapped estates are exempt and keep their mirror.

On a cluster upgraded in place, Kubernetes Dashboard is left running and Headlamp stays off until someone enables it. Nutanix recommends migrating as soon as possible. A single-instance Nutanix Enterprise AI deployment can stall worker drain: with the PodDisruptionBudget enabled, its gateway, NAI-API and PostgreSQL pods cannot be evicted. The documented fix is multiple instances of the gateway and NAI-API and a disabled budget on PostgreSQL, before the window. The declared baseline is Prism Central and AOS 7.5 or 7.6, NDK 2.3 and NAI 2.8.0 or 2.8.1.

An Endpoint That Is Now On

The MCP default deserves one plain paragraph. After the upgrade, on a connected Pro or Ultimate management cluster, the MCP server is enabled by default. It is not installed on workload or attached clusters, it is absent in air-gapped deployments, it does not expose Secrets, and it enforces Kubernetes RBAC so that each caller sees what their role permits. It also carries allow and deny rules for the resource kinds it exposes, and the release notes say to review them at upgrade time. For a multi-tenant management cluster that review is the real task: which identities the server forwards to the Kubernetes API, and which workspaces those identities can read, because an agent asking about all clusters should get an answer shaped by the same tenant boundary that shapes the UI. That is not a reason to turn it off; it is a reason to know where the bindings sit before an auditor asks.

Bare Metal and the VM Operator Are One Direction

NKP Metal is the feature that will get the headlines, and the part of it that matters for a provider is small and structural: physical hosts enter the platform through a host inventory that is scoped to workspaces, and node pools claim them through match rules. The workspace was already the tenancy unit for clusters and catalogs; now it is the tenancy unit for hardware too. A mixed cluster with VM nodes and a physical GPU pool is, from the tenant’s side, one cluster in one workspace.

The product update post also announces AHV virtual machine management through a VM Operator, with Linux and Windows VMs provisioned and managed through Kubernetes APIs. One thing worth stating plainly: neither the October 21 release notes nor the 2.19 documentation describes the operator, so today the product update post is the reference, and the operating details (which cluster it runs on, how it maps to Projects, what it needs from Prism Central) are not in either document.

Read together with NKP Metal, the direction is clear enough: the Kubernetes API as the operating model for everything on the cluster, physical or virtual. Both are the part of 2.19 that still needs a lab under it, so they stay here as a direction and come back as an article when they have been built and the documentation has caught up.

Arriving Ready

The order, for an estate on 2.18: converge projects, migrate catalogs to OCI, generate the registry token, fix single-replica NAI topologies, upgrade Kommander and the management cluster from the CLI, then upgrade workload clusters one at a time on whatever schedule each tenant can absorb. Nothing on that list is hard. All of it is sequenced, and the sequence is the contract.

What an MSP gets for honoring it is the operating model the series has been arguing toward. Today a single management cluster on 2.19 carries tenant clusters on two Kubernetes minors, 1.35.x and 1.36.x, each on its own NKP version and its own upgrade schedule, with the same preflight checks and, for ClusterClass clusters, the same single command for every one of them. A tenant that cannot take 1.36 this quarter stays on 1.35 without holding anyone else back; a tenant that wants the new DRA scheduling moves first.

The tenant-per-project design from the anchor article was argued as a control plane decision on the Prism side. With 2.19 it is also the design that makes the Kubernetes lifecycle cheap, and the platform’s own upgrade prerequisites now agree with it.