From One Prism Central to SP Central

A service provider with one Prism Central can already serve its first tenants. A project per customer, the customer’s people in an identity provider whose groups map onto roles inside that project, and nobody with a global role: that is a tenant, and when SP Central, the control plane Nutanix built for service providers, onboards a tenant, it creates a project of exactly that kind. What the small provider lacks sits around that object. Every tenant is assembled by hand, and the console the tenants need is the same one that administers the whole platform.

Portico is the open-source appliance I built for that space, and its 0.3.0 release came out on November 1. It puts one authenticated entry point in front of Prism Central, creates a tenant in one action with the same construct SP Central uses, and since this release routes the tenants’ addresses to the provider’s own router. The previous piece in this series argued that the tenant exists long before the control plane that industrializes it. This one is about the time in between, when the tenants exist and the control plane does not yet.

Everything Is Already in Prism Central

In September I wrote the manual version down, as a design for making a single Prism Central multi-tenant and putting it on the Internet. It has seven steps, and apart from an identity provider and a reverse proxy none of them needs anything Prism Central does not already have:

1. Connect Prism Central to an identity provider and test a federated login from inside.
2. Create one project per tenant, with its networks, its images and its quotas.
3. Map the identity provider’s groups onto roles inside each project, and give no tenant a global role.
4. Sign in as a tenant user and check what that user cannot see: clusters, hosts, other projects, other tenants’ images.
5. Put a reverse proxy in front, close port 9440 to everything except the proxy, and refuse local logins on the public side.
6. Test the gate from outside.
7. Write down how a tenant is added, because that procedure is what will need automating.

Each step is a click path in Prism Central or a few lines of proxy configuration, and the first tenant gets all of them with full attention. By the third, the procedure lives in someone’s memory, and the step most likely to be skipped is the visibility check, because skipping it breaks nothing anyone can see.

Exposing a Console Is Different From Publishing It

The reverse proxy is the optional step, and the one that decides whether Prism Central is exposed or published. Without it, tenants reach Prism Central directly, which puts the whole Prism Central on the Internet: its login page, its APIs and its local accounts, admin included. Local accounts are how an administrator gets in when the identity provider is down, a property every platform needs and the precise reason that door must never face outward. Exposing Prism Central leaves that door on the street. Publishing it means a tenant reaches the console only through a federated login, and nothing else gets through at all.

My first gate was a handful of nginx rules. The login form was closed for everyone, and on the API a Basic authentication header for admin always starts with the same eight characters, YWRtaW46, whatever the password, because admin: is six bytes and base64 turns every three bytes into four characters, so one prefix match blocked the administrator’s API logins. That rule knew one account: any other local account could still use Basic authentication, and the variant that refused every Basic header stayed commented out, for the tenants that call the API from outside. The first day of measurements took that design apart. On pc.7.6 the local login is a call to the identity API that carries its request id in a cookie, Basic authentication works on four generations of the API, and local login keeps working when the identity provider is down. A filter that names known paths has to be derived again after every upgrade, and it still lets through whatever it did not know about.

So Portico turns the rule around. Its proxy lives inside its own binary and never reads the username: toward Prism Central, every Authorization header, every Prism Central API key and every known local-login path gets a 403, for admin and for any other local account, with or without a session. Everything else bound for Prism Central needs a Portico session: no session, no request to Prism Central, whatever the identity provider is doing. The list of local-login paths still has to be derived again after every upgrade, but it no longer stands alone: without a session, whatever it missed is out of reach. The tenant user signs in once, on Keycloak, the open-source identity server that runs on the appliance, and Prism Central trusts the same server through SAML, so two protocols cost the user a single login, and a browser that arrives without one is sent to that login. From outside, curl -u admin:... gets a 403 from Portico before Prism Central ever sees it, and platform administration stays on an internal listener that the provider must keep off the Internet, as the install guide requires.

A Tenant Becomes One Action

The identity provider and the gate are set up once, on Portico’s setup page. The work repeated for every tenant is now one action in its admin interface, apart from quotas and images, which are not part of it yet. Portico creates the tenant’s group in Keycloak, then on each site where the tenant gets resources a Projects 2.0 project with its resource group, a SAML group and a role membership scoped to that project, and it closes each site with a visibility check that reads the project back and asserts that the group holds nothing outside it. The check reads the configuration instead of signing in as the user, as the manual step did, but it runs every time. In the first acceptance a tenant arrived in 77 seconds and seven recorded steps, and in later runs in 11; its first user signed in once, changed the temporary password and saw only that tenant’s project. Disabling the tenant locks its users out at their next login, while a session already open ends at Prism Central’s own limits. Disconnecting it from a site is refused while its project still holds anything of the tenant’s, so Portico never removes a tenant’s virtual machines.

The construct is deliberately the one SP Central uses: a project per tenant, group-based role mapping, no global roles. A provider that adopts SP Central later should find tenants it recognizes, which is why I think of Portico as a path toward SP Central. The same appliance can front several Prism Centrals, shown to tenants as sites on one public name, so a tenant with resources in two data centers signs in once, picks a site, and never sees a Prism Central address.

The Network Is the Hard Part

A tenant with a project and nothing to plug a virtual machine into is still a ticket for the network team. Since 0.2.0 a tenant also gets a ready network on each site where the provider switched it on: its own virtual private cloud with a /24, outbound access, and floating IPs, the addresses that publish its virtual machines outside the VPC, cut from a /28 of the site’s transit VPC. The layout follows the routed shared transit of Nutanix’s NVD-2219, the shared transit model this series has already walked through: the provider builds one transit VPC per site, once, and Portico builds and removes each tenant’s share of it.

Those addresses are only useful if the provider’s network knows where they live. Until 0.3.0 that was the provider’s problem: a static route on its router, typed by hand, or a BGP gateway it built itself. 0.3.0 makes a BGP session the requirement: BGP is the protocol two networks use to tell each other which addresses sit behind them, run here between the site and the provider as external BGP. The static route stays as the manual way, and a site built before 0.3.0 keeps it until the provider picks a peer. Portico builds a BGP gateway next to the transit VPC and a session that advertises the site’s routed prefix, and the peer on the other side is one of two. It can be the provider’s own router, and the Networking card then shows the three lines that router needs. Or, when the router cannot peer yet, it can be an edge router Portico builds itself: a small virtual machine running FRR, with no user, no password and no SSH, behind one static route on the provider’s core.

The two peers are also a path, and it is the part of 0.3.0 I like most. A provider whose network team has not configured BGP yet starts with the edge router and that one static route. When the router is ready, “Move to your router” transfers the session in one recorded operation, the edge stays off as a fallback, the static route on the core goes, and “Delete the edge router” removes the virtual machine. The network grows into BGP at the provider’s pace.

What the Acceptance Taught

The acceptance of 0.3.0 found the bug I learned the most from. During that acceptance, Prism Central answered some calls with timeouts in waves of two to four minutes, which is exactly the condition an orchestrator has to survive, and three things went wrong in a row. The worst: a move to the router recorded “router” as the peer before its first step, failed on a read, and “Delete the edge router” then trusted that record and deleted the live peer. The other two were of the same kind: a mark Portico had set itself was taken as a final state, and a transient answer was about to be taken as an outcome. In all three Portico had kept its own version of the truth and acted on it.

The fix is now a decision record: the stored state follows Prism Central. The peer changes in Portico’s records only inside the step that changed it in Prism Central, after that step succeeded. Every operation ends by reading the session’s state back, and a transient error is never an outcome: the waits ask again within their limit, and past it the step fails and stays on the card until Repair runs it again. It is the same principle that keeps Portico from turning into a second Prism Central: it orchestrates, and what it stores about Prism Central’s objects follows Prism Central.

How It Was Built

Portico took sixteen days from its design document to 0.3.0, and the method is the part I would reuse elsewhere. The design’s hypotheses were measured on the test clusters before the code that depended on them was final, and each measurement is a write-up that says which version it was taken on and how to take it again after an upgrade: thirty-nine so far, kept private because they describe the test environment. The code is written with Claude Code, with me as the architect, and the repository says so in its commits.

Portico’s part of all this was measured on two single-node test clusters running pc.7.6, with no production load; the move to SP Central has not been tried. The release notes list the known limits before anyone puts Portico in front of customers. The one a network team will ask about first: the edge router is a single virtual machine, so replacing it with a new release leaves the routed prefix unreachable for under two minutes, in a window the provider picks.

The Discipline, Written Down

The previous piece closed on the discipline inside the project as the one part of the design with no rollback. Portico is that discipline written as code: the visibility check that nobody skips, the role that is never global, the console that is published and never exposed. When the provider outgrows it, the tenants are already built from the construct SP Central uses.