The Firewall Is a Waypoint, Not the Gateway
Native path steering inserts the firewall, so the workloads keep their gateway and never know it is there.
A few months ago I wrote a small series about saving public Floating IPs on Nutanix. The most interesting part was the Offroad Way, where I used pfSense inside a VPC to publish multiple services behind a single Floating IP.
That design worked, but it had a clear limitation: the default gateway had to be changed inside the guest, so the machines pointed at pfSense instead of the VPC subnet gateway. That was manageable for static VMs. On NKP it was not an option, because nodes are created from images, attached to VPC subnets, and configured through IPAM, and forcing a different gateway into that lifecycle means workarounds I would not bring into a production environment.
During a datacenter PoC I tested a different approach with Flow Virtual Networking reroute policies. The goal was practical: keep the workload subnet behind the VPC virtual router, send selected traffic through pfSense, and avoid changing the default gateway inside the guest OS.
This piece belongs to the series on multitenancy as a control plane problem, and it applies the same thesis to the data path. In a tenant VPC, the routing decision belongs to the VPC virtual router, not to an appliance attached to the tenant LAN. The firewall is a service the path crosses, not the authority that defines the path. Everything that follows is that principle turned into subnets, routes, and policies.
The design worked, but only after one important adjustment. pfSense must not sit directly on the tenant LAN.
Reroute Is Policy-Based Routing
The right way to think about reroute is Policy-Based Routing inside the VPC. A Flow Network Policy can match traffic and send it to a specific Layer 3 endpoint instead of letting the VPC virtual router follow the normal route.
In this design, that endpoint is pfSense. The VPC virtual router evaluates the policy, matches the traffic, and redirects it to the firewall. pfSense then handles the packet based on its own firewall, NAT, and routing configuration.
This is not Flow Network Security Service Insertion, and it is not traffic mirroring. Service Insertion is a different feature with a different policy model. Traffic mirroring copies packets for visibility. Reroute changes the forwarding path.
It is also worth being precise about the gateway terminology. In Flow Virtual Networking, Network Gateway has a specific meaning related to connectivity services such as VPN, VTEP, or BGP. The element that evaluates the policy here is the VPC virtual router, not a Network Gateway. This is inline path steering through PBR.
The distinction matters during troubleshooting. In a reroute design I look at routing, policy match, return path, and firewall routes before I look at security policy behavior.
Why pfSense Should Not Be on the Tenant LAN
The first iteration of the PoC looked obvious: attach pfSense to the tenant LAN, add a WAN interface, attach a Floating IP, and use a reroute policy. That is close to how many traditional firewall designs are built.
Inside a Nutanix VPC reroute design, that creates a problem. If pfSense has an interface on the tenant LAN, that subnet becomes directly connected from the firewall point of view. When return traffic comes back through the WAN, pfSense already knows how to reach the VM directly and has no reason to send the packet back through the VPC virtual router.
VM
-> VPC subnet gateway
-> reroute policy
-> pfSense
-> WAN
-> Floating IP
-> external network
-> Floating IP
-> pfSense WAN
-> direct return to VM because the tenant LAN is connected
That is an asymmetric path. The outbound traffic was steered by the VPC virtual router, but the return traffic bypasses it because pfSense treats the tenant subnet as local.
The fix is to not connect pfSense to the tenant LAN at all. pfSense should be a routed service endpoint, not the default gateway or the directly connected router for the workload subnet.
The Transit Subnet
The working design uses three subnets with separate roles. The tenant subnet hosts the workloads. The transit subnet is the handoff between the VPC virtual router and pfSense. The WAN subnet is where the pfSense external interface lives, with the Floating IP attached to that interface.
This is not a Nutanix Transit VPC. Here, transit subnet simply means a small service subnet inside the VPC, used only to connect the VPC virtual router to the firewall appliance.
The two firewall-facing subnets are deliberately point-to-point /30 links, one for the inside handoff and one for the outside. Together they give pfSense the inside and outside legs a perimeter firewall expects, so the appliance behaves like a normal edge firewall while remaining a routed service inside the VPC, never attached to the tenant LAN. The small size is the point. Nothing lives on these segments except the two ends of each handoff.
Tenant subnet
Tenant subnet
172.20.60.0/24
VPC subnet gateway
172.20.60.1
Example VM
172.20.60.20
Transit subnet
Transit subnet
100.65.0.0/30
VPC subnet gateway
100.65.0.1
pfSense transit interface
100.65.0.2
Firewall WAN subnet
Firewall WAN subnet
192.168.200.0/30
VPC subnet gateway
192.168.200.1
pfSense WAN interface
192.168.200.2
Floating IP
203.0.113.10
pfSense has two interfaces in this design: one on the transit subnet and one on the WAN subnet.
pfSense transit
100.65.0.2/30
pfSense WAN
192.168.200.2/30
It does not have an interface on 172.20.60.0/24. That is the relevant detail. The tenant subnet stays behind the VPC virtual router, and pfSense reaches it through a static route.
Route on pfSense
172.20.60.0/24 via 100.65.0.1
The default route on pfSense still points out through the WAN side.
Default route on pfSense
0.0.0.0/0 via 192.168.200.1
With this layout the routing model is explicit. The VPC virtual router owns the subnet gateway IPs, pfSense owns the firewall and NAT behavior, and the transit subnet connects the two without making pfSense part of the tenant LAN.
Why I Use 100.65.0.0/30
In this example the transit subnet is 100.65.0.0/30. This prefix belongs to the RFC 6598 shared address space, commonly associated with CGNAT. Here it is not used for carrier-grade NAT. It is used as a service-only transit range.
The reason is practical. The transit subnet is not a tenant network, and no workload should live there. It is only a technical handoff segment between the VPC virtual router and pfSense. A small range from the shared address space makes that role easy to recognize and keeps it visually separate from normal RFC 1918 tenant addressing.
In production the selected prefix still has to fit the enterprise addressing plan and must not overlap with tenant networks, VPN prefixes, advertised routes, or other uses of the same shared address space. The exact prefix is not the important part. The important part is that the transit subnet is a service path, not a workload network.
Blocking Direct Access to Firewall Subnets
After creating the transit and WAN subnets I added explicit deny policies from the tenant subnet to both firewall-facing networks. Workloads should not talk directly to the pfSense transit or WAN interfaces.
172.20.60.0/24 -> 100.65.0.0/30 Deny
172.20.60.0/24 -> 192.168.200.0/30 Deny
This stops a workload from treating those subnets as normal destinations. The workload subnet should only use its default gateway. The reroute decision belongs to the VPC virtual router, not to the VM.
Scenario One: Internet Inspection and Inbound NAT
The first scenario is Internet inspection through pfSense. The Flow Network Policy matches traffic from the tenant subnet to external destinations and reroutes it through the firewall.
Source
172.20.60.0/24
Destination
External
Action
Reroute
Incoming reroute IP
100.65.0.2
Outgoing reroute IP
192.168.200.2
Fallback action
Drop
Reverse direction
Enabled
I prefer Drop as the fallback action here. If the firewall service IP is not reachable, traffic should not silently bypass inspection.
In practical terms the tenant VM still uses the VPC subnet gateway as its default gateway. The VM does not need to know that pfSense is in the path. The full path is shown below.
The Floating IP is attached to the pfSense WAN address. When pfSense sends traffic out, the platform translates the private WAN address to the Floating IP. Return traffic follows the reverse path through the Floating IP, reaches pfSense on the WAN, and is routed back to the tenant subnet through the transit interface.
Where inbound NAT fits
Inbound NAT still makes sense in this design, but for a different reason than in the Offroad Way. There, NAT was part of how multiple services were published behind a single Floating IP. Here, NAT is simply the publishing function of the firewall.
An external client reaches the Floating IP. pfSense receives the traffic on its WAN interface. A DNAT rule forwards the connection to a VM inside the tenant subnet.
external client
-> Floating IP 203.0.113.10
-> pfSense WAN 192.168.200.2
-> DNAT to VM 172.20.60.20
-> route to 172.20.60.0/24 via 100.65.0.1
-> VPC subnet gateway
-> VM 172.20.60.20
The return path is the relevant part, and it uses two different mechanisms. Inbound, pfSense delivers the translated packet to the VM through its static route toward the VPC subnet gateway on the transit subnet. Outbound, the VM answers the external client through its own default gateway, and the reverse direction of the reroute policy brings that response back to pfSense, so the NAT state holds. NAT publishes the service, the transit subnet preserves the routed path, and the VM default gateway never changes.
Scenario Two: Selective VPN Routing
The same pattern is useful when pfSense is not the default Internet inspection point. A common example is a site-to-site VPN.
Assume pfSense terminates an IPsec tunnel to a customer network. You do not want all traffic from the tenant subnet to cross pfSense. You only want traffic for the customer prefix to use the VPN.
Instead of matching all external destinations, the Flow Network Policy matches only the remote customer network.
Source
172.20.60.0/24
Destination
10.200.0.0/16
Action
Reroute
Reroute IP
100.65.0.2
Fallback action
Drop
The logic is straightforward. For 10.200.0.0/16, traffic goes through pfSense and into the IPsec tunnel. For everything else, it follows the normal VPC path straight to the Internet, without touching the firewall.
Drop matters even more here than in the inspection scenario. In Scenario One a permissive fallback means traffic skips inspection. In Scenario Two it means something worse. If pfSense is unreachable and the fallback is permissive, traffic for 10.200.0.0/16 does not just lose connectivity. It falls back onto that same default route and leaves over the public Internet instead of the IPsec tunnel. That is customer traffic escaping the VPN, not a simple outage, so the fallback must be fail-closed.
One caveat worth stating plainly. The Flow fallback governs only the reroute decision, so it covers the case where pfSense is unreachable. It does not cover a tunnel that is down while pfSense is still up. At that point the packet has already reached the firewall and pfSense routes it by its own table, where the default route still points out the WAN. Customer traffic can leave over the Internet anyway, unless pfSense itself drops 10.200.0.0/16 when the tunnel is down, with a reject route or a matching firewall rule. Fail-closed has to hold at both layers, not just in the Flow policy.
Note that this scenario uses a single reroute IP, not two. The inspection design needs both the inside and the outside handoff because traffic enters pfSense on transit and leaves on WAN. The VPN design only needs the inside handoff, because the outside leg is the IPsec tunnel that pfSense owns directly.
Here reroute is not used to inspect all Internet-bound traffic. It steers one specific prefix toward the firewall appliance that owns the VPN tunnel. The VM keeps the VPC subnet gateway as its default gateway, pfSense owns the VPN, and the Flow policy decides which destination prefix should use that path.
How This Differs From the Offroad Way
The Offroad Way made pfSense part of the addressing and NAT design. That was useful when the goal was to reduce Floating IP consumption and publish multiple services behind a single public address.
This design solves a different problem. pfSense is not the tenant gateway. It is a routed service endpoint reached through a policy decision on the VPC virtual router.
With the Offroad Way the workload had to know about pfSense: the guest used the firewall as its default gateway, and services were published through NAT. With reroute the workload keeps the VPC subnet gateway as its default gateway, and the VPC virtual router decides when pfSense enters the path. That makes the design more suitable for dynamic workloads, Kubernetes nodes, tenant environments, and any case where guest-level routing changes are undesirable.
Operational Notes
Keep the policy scope precise. If the policy is too broad you can match traffic that belongs to the firewall path itself and create loops or unexpected behavior. The tenant subnet should be the source, and the transit and WAN subnets should never be treated as tenant destinations.
Make the pfSense routes explicit. The firewall has to know that the tenant subnet is reachable through the VPC subnet gateway on the transit subnet. Use separate reroute IPs for two-legged designs: the transit interface is the inside handoff, the WAN interface is the outside handoff.
Treat the fallback action as a security decision, not a default. Drop is the safer choice, but validate how your Prism Central version evaluates reachability of the reroute target before you rely on it as fail-closed. The behavior on an unreachable next-hop is the part worth testing, not assuming. And the policy fallback is only the first layer. What the appliance does with steered traffic when its own egress is down, the tunnel-down case from Scenario Two, is the second, and both have to hold.
Plan for failure of the appliance itself. A single pfSense in the path is a single point of failure for every flow it inspects or tunnels. pfSense HA with CARP across VPC subnets and Floating IPs is its own design exercise and is out of scope here, but it belongs on the list before this becomes a production standard.
Watch the MTU. The extra hop, and especially the IPsec tunnel in the VPN scenario, reduces the effective path MTU. Account for it and apply MSS clamping where needed, or you will chase intermittent failures that look like anything but routing.
The VPC Stays the Routing Boundary
One design choice deserves to be stated as a rule. Do not attach pfSense to the tenant LAN unless the goal is specifically to make pfSense the tenant gateway. That single attachment changes the entire return-path model described here.
The transit subnet is the part that makes this work. Without it, pfSense sees the tenant LAN as directly connected and the return traffic skips the VPC virtual router. With it, the VPC stays the routing boundary, pfSense stays a routed service endpoint, and the reroute policy decides when traffic crosses the firewall. That separation is what makes the design safe to apply to dynamic workloads where guest-level routing changes are not an option. And in an MSP design it is what makes the pattern repeatable: the same transit layout, the same policies, stamped into every tenant VPC, with the routing decision held by the control plane instead of renegotiated inside each appliance.