)
If Part 1 explored network sovereignty—the ability to explain traffic paths and security policy—Part 2 turns to validation sovereignty: the ability to explain code changes and the quality criteria behind them. For an operator running its own cloud platform, sovereignty depends not only on which changes enter the operational baseline, but also on how those changes are validated before they do.
As kt cloud operates KCP (kt cloud PLATFORM), it has been building and operating its own downstream distribution, “KTC OpenStack,” based on the OpenStack community version. In this context, building on community OpenStack requires more than installing software. It means directly building and operating the software supply chain that spans code, images, and charts. Before changes to these components are reflected in production, a validation system is essential.
This article explains why kt cloud moved beyond conventional CI toward a Zuul-based gating model, what limitations it found in earlier validation approaches, and how it designed downstream OpenStack validation around upstream validation assets.
The Operational Responsibility of Running OpenStack In-House
Adopting community OpenStack as the core platform involves more than a software selection decision; it represents a shift in operational responsibility. When using a commercial distribution, the vendor shares much of the responsibility for distribution, patching, validation, and incident response. Choosing to use the upstream community version directly, however, means the operator must build and continuously maintain the entire workflow, from code management, builds, patches, and validation to deployment.
At first, the simplest approach is to stay as close as possible to the upstream configuration. However, as the platform grows, image builds, packaging work, driver modifications, and new feature development required for the operating environment begin to accumulate. As a result, upstream OpenStack and downstream changes tailored to the kt cloud environment come to coexist.
kt cloud organized this structure as KTC OpenStack. But the coexistence of upstream and downstream code alone does not turn operational principles into practice. The merge-time validation path also had to define which changes are allowed, how regressions in existing functionality are blocked, and when local changes should be contributed back upstream.
KTC OpenStack manages downstream divergence through three rules:(1) use upstream OpenStack as the default baseline,(2) implement only essential features or major bug fixes that are not available upstream, and (3) contribute internally developed changes back upstream whenever possible. The third rule prevents local changes from becoming permanent private assets and helps return them to a shared baseline. Gating verifies compatibility and regression risk before, so that these rules do not remain simply declarative.
Why Gating, Rather Than CI Alone?
kt cloud already had several options for running validation, including Jenkins, GitHub Actions, and custom script-based pipelines. The harder problem lay in maintaining two quality baselines at once. Upstream jobs capture the supported combinations and regression criteria used by the OpenStack community, but they do not automatically cover kt cloud’s OVN version, Kubernetes-based control plane, or downstream extensions. Internal tests can cover those local combinations, but relying on them alone risks drifting away from upstream quality criteria. kt cloud therefore needed a model that could inherit upstream validation assets while adding targeted checks for KCP-specific combinations where necessary.
The quality baseline was equally important. If downstream code cannot pass upstream jobs, the failure may indicate more than a local test gap; it may reveal a regression in existing OpenStack functionality. Even when internal patches and features are added, the standard should go beyond “it works in our environment.” It should also ask whether the change passes the validation criteria defined upstream. For this reason, kt cloud chose a gating system that evaluates changes before merge, instead of a CI setup that simply runs tests after the fact. The goal was to connect to OpenInfra validation assets, reference them, and inherit them as part of downstream validation.
Figure 1. 5-Stage Multi-Layer Gating Pipeline
Figure 1. shows the multi-layer validation scope targeted by the KTC Gating System. Rather than judging a change with a single test, the pipeline expands validation step by step: from static analysis, to cloud configuration, to functional and regression validation, and finally to container deployment validation. The Zuul-based architecture described next provides the control plane used to execute and govern this scope.
The KTC Gating System pipeline starts with Python unit tests and Tox-based static analysis, then expands its operational scope around DevStack-based OpenStack test environment provisioning and Tempest-based functional and end-to-end validation. Together, Rally-based performance and regression validation and Helm/GeneStack-based Kubernetes deployment validation form an extension layer for validating the full supply chain from code to images and charts.
Zuul-Based KTC Gating System
kt cloud chose Zuul because it lets downstream validation follow the same gating flow used by the OpenStack community. The key requirement was direct alignment with validation assets that the community continuously evolves. Zuul is an open source project gating system that runs jobs before changes are merged, helping ensure that only passing changes are accepted. It also supports cross-project validation and reusable Ansible-based jobs, both of which are important when a downstream distribution needs to validate related changes across repositories.
To support this model, the KTC Gating System was designed around two axes. The first is the Zuul CI cluster, which acts as the control plane for managing pipelines, job definitions, queues, and validation logic. The second is the OpenStack-based resource provider, which supplies the VMs and test environments where jobs actually run. In effect, the system uses Zuul and OpenStack to validate OpenStack again. The control plane runs in the standard KCP (kt cloud PLATFORM) Kubernetes environment, while the resource provider dynamically supplies test resources from OpenStack IaaS.
Figure 2. KTC Gating System Architecture
Figure 2. shows the boundary of responsibility between the control plane and the execution layer. In the control plane running on the KCP standard Kubernetes environment, Zuul Scheduler coordinates pipelines and queues, ZooKeeper shares distributed state, and Nodepool requests test nodes from OpenStack. Argo CD manages the deployment state of these components. The OpenStack-based resource provider, which serves as the execution layer, dynamically provisions ephemeral VMs with KVM nested virtualization on demand and provides an isolated test environment for each validation job.
The next key design choice was upstream job inheritance. In KTC OpenStack validation, upstream test environment configuration, execution procedures, and Tempest-based validation flows are reused, while only the variables required for the downstream repository and target release are declared. For example, ktc-neutron, kt cloud’s downstream Neutron repository, inherits the upstream Neutron job as its parent and specifies the ktc-neutron repository in required-projects and devstack_plugins.
– job:
name: ktc-neutron-tempest-plugin-ovn-2025-1
parent: neutron-tempest-plugin-ovn-2025-1
required-projects:
– name: openstack/devstack
override-checkout: stable/2025.1
– name: openstack/requirements
override-checkout: stable/2025.1
– name: kt-cloud-stack/ktc-neutron
vars:
devstack_plugins:
neutron: https://github.com/kt-cloud-stack/ktc-neutron.git
In this example, parent specifies the upstream Neutron job that serves as the validation baseline, while required-projects and devstack_plugins replace only the actual validation target with ktc-neutron. In other words, the validation target and release baseline are declared without duplicating the validation procedure.
Figure 3. Upstream Job Inheritance & Downstream Patch Flo
Figure 3. shows the flow in which the environment configuration and validation procedures defined by the upstream parent job are reused, while only changes in KTC repositories such as ktc-neutron and ktc-nova are connected as downstream validation targets. This makes it possible to repeatedly confirm, before merge, that internal patches remain compatible with existing OpenStack functionality.
The advantage of this approach is that downstream teams can reuse upstream jobs as the quality baseline without rewriting the validation procedure. kt cloud declares which upstream job to inherit and which downstream code and release baseline to run it against. When an upstream job definition changes, that change can be reviewed and reflected through the connected repository and branch configuration, reducing divergence compared with copying and maintaining test logic separately. Zuul runs the required jobs before changes are merged and allows only passing changes to proceed. Internally, kt cloud prioritized detecting regressions in existing functionality before validating new features by applying a simple principle: “upstream jobs must pass.”
Inherited jobs run in the following sequence before a change is merged. While Figure 1 shows the full validation scope from static analysis to deployment, the table below explains how one OpenStack functional validation job receives resources, prepares the code, runs DevStack and Tempest, and preserves the results. Each execution result and log remains available as evidence for merge decisions and follow-up improvements.
| Stage | Execution details |
| 1. Environment preparation | Provision test VMs from the OpenStack resource provider. |
| 2. Code preparation | Check out ktc-neutron and the required projects on the specified branches. |
| 3. Cloud configuration | Use DevStack to configure an OpenStack test environment based on downstream Neutron. |
| 4. Validation execution | Run the Tempest jobs defined by upstream to check functionality and regressions. |
| 5. Result preservation | Collect logs and test results for merge decisions and follow-up analysis. |
As a result, the focus shifted from “operating our own separate CI” to “connecting OpenInfra community validation assets to our own quality baseline.” This implementation also helped internalize the development and validation culture of OpenStack into kt cloud’s operating model.
Keeping Quality Criteria and Validation Under Operator Control
In this system, validation functions as part of the platform’s control capability. An operator may be able to modify platform components, but that authority becomes operationally meaningful only when each change can be independently validated against the combinations used in production. The KTC Gating System closes that gap by connecting the ability to understand and modify code with evidence that each change still meets the operational baseline.
The most important lesson from building the Zuul-based KTC Gating System was this: reference validation assets instead of copying them. Simply duplicating upstream jobs and test code into downstream may look easier to control at first, but over time it increases maintenance cost and the risk of inconsistency. A design based on inheritance and reference, by contrast, can bring upstream changes themselves into the validation system.
The second lesson is that gating turns downstream policy into merge rules. KTC OpenStack adopted the principle of using upstream as the default baseline, maintaining only the changes that are necessary, and contributing those changes back upstream whenever possible. Principles alone, however, can vary under schedule pressure or exception handling. When required jobs must pass as a merge condition, the quality baseline becomes part of the change path itself, replacing individual judgment or after-the-fact review.
The third lesson is that quality validation is also part of what makes a cloud operationally sovereign. Operating infrastructure directly does not end with owning servers and networks. Operators must be able to explain how code enters the platform, by what criteria it is validated, and which changes are reflected in production. The Zuul-based KTC Gating System provides a foundation for tracing this process by connecting changes, validation results, and merge decisions into a repeatable flow.
The KTC Gating System is still in its early stages, and its current scope is focused on OpenStack code validation. kt cloud is not yet presenting it as a system that has already blocked production regressions in operation. With that limitation clearly stated, the next goal is to expand validation from individual projects to an integrated environment that more closely resembles the KCP architecture.
The scope of the KTC Gating System will also expand in stages. First, kt cloud will continuously reflect changes in upstream OpenStack releases and jobs so that downstream code can keep pace with the community’s latest quality standards. Next, beyond validating individual projects, kt cloud plans to automatically configure clusters on real VMs that are closer to the KCP architecture and to supplement platform integration and operational requirements that are difficult to cover with upstream jobs through KCP-specific jobs. Even as the validation scope expands, the principle remains the same: community validation assets should serve as the baseline, while kt cloud’s own criteria should be explicitly added only where necessary, so that the validation process and results remain explainable.
Conclusion
The Zuul-based KTC Gating System connects OpenStack community validation assets to downstream operations. A Kubernetes-based control plane coordinates validation policy and execution flow, an OpenStack-based resource provider supplies isolated test environments, and upstream job inheritance applies community quality standards to downstream changes. For kt cloud, this is another dimension of sovereign cloud operations: choosing and operating the technology stack directly, while also being able to explain by what criteria code changes are validated and what quality baseline they must pass before reaching production.
Author bio: Geonhee Kim is responsible for building the OpenStack-based quality validation framework and Zuul-based Gating System for the KCP platform at kt cloud.
- Part 3: From CI to Gating — Validating Downstream OpenStack with Zuul - October 28, 2026
