Part 2: Building VPC Networking with OpenStack Neutron and OVN

The first pillar of the sovereign cloud operating model described in the Prologue is the ability to directly understand and control both the network control plane and the execution plane. This article explains how kt cloud restructured OpenStack Neutron’s open resource model into a VPC service for KCP (kt cloud PLATFORM) and implemented Multi-AZ networking using OVN as the distributed execution plane. It then examines the ACL ownership issue identified during the integration with Kube-OVN, kt cloud’s experience in sharing the issue with the upstream community, and the approach used to establish auditability through VPC Flow Logs.

ACL Ownership Challenges in a Shared OVN Environment

For a cloud service provider, networking is more than a connectivity layer. Customer traffic paths, isolation policies, security logs, and failure analysis points all converge in this layer. Operators therefore need to understand, automate, and explain network behavior in an auditable way.

While building KCP, kt cloud implemented VPC, Security Group, NAT, and Flow Logs capabilities on top of OpenStack Neutron and OVN. In addition to Neutron’s default Security Group model, kt cloud added a stateless NACL feature that applies policy at subnet boundaries. KCP was integrated so that Neutron and Kube-OVN share a single OVN Northbound DB. In that configuration, kt cloud discovered a regression in which Neutron-created Security Group ACLs were deleted when the Kube-OVN controller restarted.

kt cloud reproduced the behavior in a shared OVN environment and confirmed that ACLs created by Neutron were being included in the Kube-OVN controller’s reconciliation cleanup scope. Based on this analysis, kt cloud proposed in Kube-OVN upstream issue #5995 that each ACL should carry origin and controller ownership information in external_ids, and that each controller should limit garbage collection to the objects it owns. This case illustrates how an operational issue found in production can be turned into a reproducible problem statement and an upstream improvement proposal around ownership boundaries.

Project	Contribution	Link
Kube-OVN	Shared an issue about preserving Neutron Security Group ACLs in a shared OVN environment	#5995

This issue showed that when two cloud management systems share the same OVN Northbound DB, each controller’s reconciliation and garbage collection scope must be explicitly bounded. In a shared database, operators must define not only which resources a controller can create, but also which resources it is allowed to modify or delete.

The key point in this regression was not that reconciliation itself was broken, but that there was no provenance to distinguish where each object came from. When multiple controllers share one database, the boundaries for modification and deletion must be defined alongside creation permissions. Even if each controller behaves correctly from its own perspective, a garbage collection scope that is broader than its ownership boundary can remove valid objects managed by another controller.

Understanding this case first requires examining how OpenStack and Kubernetes share a single OVN fabric in KCP. The next section explains how the two platforms use the same network foundation while maintaining different resource responsibilities. 

One OVN Fabric, Two Consumers

KCP is a cloud-native IaaS architecture that combines Kubernetes and OpenStack. OpenStack Nova orchestrates VMs, Kubernetes orchestrates containers, and Neutron and Kube-OVN are configured to share the same OVN foundation. However, sharing the same fabric does not automatically create direct VM-to-Pod connectivity. A consistent communication path requires non-overlapping address spaces, connections between Logical Routers and Logical Switches, routing and NAT policies, and clear ownership boundaries for the ports and ACLs managed by each controller.

Consumer	Role	OVN Resource Mapping
OpenStack Neutron	VM networking (VPC, Subnet, Router, Security Group)	Logical Switch, Logical Router, ACL
Kube-OVN	Pod networking (CNI)	Logical Switch, Logical Switch Port

A core operational principle was to distinguish each system’s source of truth from the transformed state in the OVN Northbound DB. The Neutron DB and Kubernetes CRDs represent the original desired state managed by each controller, while the OVN Northbound DB is the shared observation point where the logical network state generated by Neutron and Kube-OVN converges. In KCP, Logical Routers created by Neutron and Logical Switch Ports created by Kube-OVN are connected on the same OVN fabric. By combining non-overlapping address spaces with routing and NAT policies, KCP implemented VM-to-Pod communication paths without relying on a separate centralized virtual router or firewall appliance.

This architecture requires clear operational boundaries. If each controller’s authority to create and clean up resources is not clearly separated, unintended conflicts can occur on shared resources.

Figure 1. KCP VPC Reference Architecture

Figure 1 shows how OpenStack Neutron and Kube-OVN share a unified OVN fabric in KCP for VM and Pod networking, Multi-AZ routing, and Gateway Chassis-based external connectivity. It highlights the shared network foundation while preserving the distinct resource responsibilities of each controller.

OVN-Based VPC Architecture and Operational Enhancements

From vDOM to an OVN-Based VPC

Neutron does not provide a single object that represents a VPC in the AWS sense. KCP combines project, network, subnet, router, port, and Security Group resources into one customer-facing service model. To customers, a VPC is a lifecycle unit for managing an isolated address space together with routing and security policies. Internally, it is implemented as a set of OVN Logical Routers, Logical Switches, DHCP settings, NAT rules, Port Groups, and ACLs. The service layer coordinates creation order, partial failure handling, quotas, and deletion workflows across these resources. This separation lets KCP preserve the customer-facing VPC experience while independently evolving the internal topology and policy implementation. It also means operators must continuously validate and maintain the orchestration layer itself.

Service Concept	OpenStack/Neutron View	OVN Implementation View
VPC	An isolated network space and related resource set within a project.	A combination of Logical Routers and multiple Logical Switches
Subnet	IP range, gateway, and DHCP policy	Address space and DHCP settings within a Logical Switch
Router	Inter-subnet routing and external connectivity	Logical Router, Static Routes, and NAT rules
Security Group	Stateful security policy at the port level	Port Groups and ACLs
NACL/FWaaS	Policy control at subnet or router boundaries	Boundary policies based on Port Groups and ACLs

Neutron provides the resources needed to implement isolated address spaces and routing, but it does not provide a VPC object that customers can create, apply quotas to, audit, and delete as a single unit. KCP therefore had to coordinate the lifecycle of multiple Neutron resources in the service layer, including partial failure states and cleanup procedures. This distinction is not just a naming issue; it means operators must maintain a separate control layer above the API boundary.

In the previous environment, virtual firewalls (vDOMs) handled tenant isolation and routing. They were stable, but they did not meet the requirements of a software-defined cloud for horizontal scalability, Multi-AZ expansion, and API-driven automation. To address these limitations, KCP combined Neutron resources into a customer-facing VPC service model and used OVN’s distributed logical routing and ACLs as the data plane. This transition moved routing and policy enforcement responsibilities away from dedicated appliances and into a software-defined control plane with per-node enforcement state.

KCP’s VPC design separates routing responsibilities by function instead of concentrating all connectivity responsibilities in a single router. A single-router model is simple to implement and operate at first, but internet egress, NAT, external connectivity, and inter-VPC connectivity can become coupled within one change domain. A multi-router model adds complexity in router interconnection, route control, and failure diagnosis, but it separates the lifecycle and change scope of each connectivity function. It also provides a foundation for future capabilities such as VPC peering, Transit Gateway-like services, and private/public subnet separation. The core design principle is to preserve a single VPC experience for customers while separating responsibilities internally.

With the move to an OVN-based VPC architecture, KCP could use open source technologies across the OpenStack ecosystem, including Neutron, OVN, Neutron tenant networks, and Octavia. The goal was to deliver a public-cloud-like VPC experience while keeping the implementation model and failure analysis points visible and understandable to operators.

Operating Security Groups and NACLs as Layered Policies

KCP treats security policy as two complementary layers: stateful Security Groups at the VM or port level, and stateless NACLs at the subnet or router boundary. Security Groups track connection state and enforce workload-level access control, while NACLs independently evaluate directional allow and deny policies at network boundaries. Both features can use OVN ACLs as their execution mechanism, but their policy semantics and operational responsibilities are different. By placing boundary policies in NACLs and workload-specific policies in Security Groups, and by validating conntrack state together with rule evaluation order, operators can explain and trace policy conflicts in which an allow rule in one layer is effectively overridden by a deny rule in another.

Building Auditability with VPC Flow Logs

For operators to trust the network, they must be able to explain which policy allowed or denied specific traffic. VPC Flow Logs support connectivity troubleshooting, security incident response, usage monitoring, and auditability in certification environments such as CSAP. This matters because a basic 5-tuple flow record is not enough to explain policy intent or tenant context. The existing Neutron Packet Logging Framework records results either in local files on compute nodes or in the system journal depending on configuration, which can increase per-node disk I/O and distributed log collection overhead at scale. To address this, KCP used the ACL sampling capability in the OVN 24.09 series to collect ACL-matched flow information through IPFIX.

1. OVN reflects ACL and Sample settings in the logical pipeline.
   
2. The OVS IPFIX Exporter attaches observation domain and observation point identifiers to sampled flows before sending them to a Collector.
   
3. As shown in Figure 3, KCP extends the default IPFIX export flow into a distributed per-compute-node collection model.
   
4. OVS on each compute node sends IPFIX data to a local Goflow2-based Collector.
   
5. The Collector uses an OVSDB Client Cache to enrich each sampled record with tenant and interface metadata, including project, port, ACL action, and direction.
   
6. The enriched records are converted into Protobuf format and forwarded to the acl topic in Kafka.

Figure 2. OVN ACL Sampling and IPFIX Export Processing Flow: traffic matching OVN ACLs is sampled and exported as IPFIX records for downstream collection and analysis.

Figure 2. shows the basic processing flow in which traffic matching an OVN ACL is generated as an IPFIX record. The area after the Collector represents a common consumption pattern, while KCP’s actual implementation is extended as the distributed collection and enrichment pipeline shown in Figure 3. In that implementation, each compute node runs its own Collector and uses an OVSDB Client Cache to enrich tenant and interface metadata before forwarding the results downstream. Because Collectors are deployed per compute node, central collection bottlenecks can be reduced. If CPU or memory pressure appears on a specific node, that node’s Collector can be isolated and managed independently.

Figure 3. KCP VPC Flow Logs Collection and Metadata Enrichment Pipeline: a distributed per-compute-node flow collection model that enriches sampled records with tenant, interface, and ACL metadata before forwarding them to Kafka.

Operational Principles Learned from a Shared OVN Environment

Several operational principles emerged from building an OVN-based VPC in a shared control-plane environment.

Align terminology and resource mappings first. Similar concepts appear under different names—such as OpenStack Router, Kube-OVN VPC, and OVN Logical Router—so mapping these terms first makes both design work and incident response faster.

Use the OVN Northbound DB as the reference point. Compare the API desired state with the logical state in the OVN Northbound DB. The Neutron DB and Kubernetes CRDs are the respective source states, while the OVN Northbound DB is the common observation point where the transformed results from both controllers meet.

Strictly separate CIDR ranges. Centrally manage the allocation policies for Neutron tenant CIDRs and Kubernetes Pod CIDRs. If address ranges overlap, route selection in Logical Routers and NAT behavior can vary across environments. 

Design auditability from Day 1. Include policy identifiers and tenant metadata in the logging design from Day 1 so that flow records can be traced back to the Security Group or NACL decision that allowed or denied the traffic.

Closing Thoughts

OVN provided the foundation for operating VM and Pod networks on a unified distributed data plane, but network sovereignty was not achieved simply by adopting these components. KCP turned Neutron’s open resource model into an operational VPC service model, executed routing and security policies through OVN, and defined ownership and reconciliation boundaries for objects shared with Kube-OVN. It also translated an ACL conflict found during operations into a reproducible upstream issue and made policy enforcement observable through VPC Flow Logs based on ACL sampling and IPFIX. In practice, technical ownership combines the ability to design a service model, validate runtime data-plane state, explain root causes, propose fixes upstream, and extend observability to meet audit requirements.

About the authors: Moonsang Yoo and Chanyeol Yoon are Cloud Platform Engineers at kt cloud, where they focus on OpenStack-based network virtualization and cloud platform engineering for the KCP platform.