From NFV to a Company-Wide Cloud: KDDI’s OpenStack Journey with MShip3

This article is based on a Japanese article originally published as series of KDDI’s Tech-note.

Cloud Migration and Virtualization at KDDI

KDDI has been implementing a unified private cloud across the entire company since fiscal year 2023. First, let me briefly introduce the background leading up to the introduction of these clouds.

Virtualization and Cloud Migration in the Enterprise Domain

For many years, KDDI has provided its KDDI Cloud Platform Service (KCPS)  to customers, and KCPS is widely used internally. Furthermore, a private cloud for the internal IT department has been in operation since around 2018. Thus, the use of proprietary and private clouds has progressed, primarily for enterprise and internal IT department purposes. In addition, the use of public clouds, led by AWS, has also progressed for web-based services and in-house systems.

Network Virtualization Trends

On the other hand, the situation was different for the virtualization of communication systems that provide mobile and fixed communications. In particular, in communication systems that require high quality, performance, and real-time capabilities, the initial server virtualization technology had significant performance overhead, which limited the adoption of virtualization technology. However, the convenience of cloud services such as AWS and the flexible software control capabilities of virtualization technology were expected to improve operability and equipment utilization efficiency in the world of communications, and there were high expectations for this. In particular, communication systems have a larger scale of equipment compared to enterprise use, and the benefits of equipment sharing were also expected. The concept of virtualization of communication systems was presented in 2012 as network virtualization: Network Function Virtualization (NFV) , and implementation towards realization progressed throughout the industry.

The consideration of NFV (Network Functions Virtualization) drove software development, and network virtualization technologies, primarily open-source software (OSS), began to mature. Around 2016, KDDI also started to implement it in practice. It was around this time that I began to become involved in this field.

Subsequently, virtualization using OpenStack, an open-source private cloud software, became the de facto standard in NFV systems. However, because NFV systems requiring high quality and performance had their own unique tuning requirements, KDDI’s virtualization progressed on a system-by-system basis, with the exception of the commonality example in the VoIP systems, I was involved in developing.

KDDI’s Unified Introduction of a Private Cloud

However, as you can imagine, the proliferation of private clouds results in high operating costs, which are passed on to us. Furthermore, these tuning issues, which were initially technically difficult, have become solvable as technology has matured. In addition, KDDI has acquired sufficient skills and knowledge to move towards standardization.

Therefore, the concept of an integrated private cloud that could meet the requirements of Network Functions (NFs) while also running enterprise systems was conceived, and construction began in fiscal year 2023 under the internal system name MShip3, with actual operation starting in fiscal year 2024 

Large-Scale Private Cloud: MShip3

The tanuki (raccoon dog) logo is our trademark.

MShip3 is KDDI’s new private cloud, which began construction in 2023, and has already become a major infrastructure in just two years since its construction.

• Number of physical servers: 4,500 or more
• Number of virtual servers: 20,000 VMs or more
• Number of vCPUs provided: 300,000 vCPUs or more
• Provided storage capacity: 50 PB or more

These infrastructures are distributed and deployed in six locations across Japan.

Using OpenStack

A private cloud centered around OpenStack and the components it uses.

KDDI’s private cloud, MShip3, is built on OpenStack, an open-source software. Users within the company who use MShip3 can access these clouds via OpenStack’s CLI/API/GUI, etc.

Instead of simply deploying a standard, off-the-shelf OpenStack, we’ve implemented a mechanism to transparently show multiple regions to end users, and made customizations and improvements to enhance usability, addressing the common“usability issues” often associated with private clouds. In particular, we’ve kept the separation to a minimum, like AWS where regions and availability zones are properly configured, so users don’t need to change their OpenStack access destinations or credential information for each environment. We’ve made it so that users only need to be aware of the endpoints for their development and production environments.

Use of Ceph

Scope of Ceph Usage

Furthermore, for storage, we adopt Ceph, a software-defined storage (SDS) solution that is also open-source software. Ceph is an SDS that can run on the Linux OS and can combine numerous SSDs and HDDs from multiple servers to present as a single large storage resource. Another major feature of Ceph is its support for various protocols, including block storage and object storage.

Furthermore, the ability to implement more detailed design allows for flexible resource allocation and redundant configurations that reflect the operator’s intentions, which is a major advantage. KDDI is already stably operating over 450 Ceph storage units across more than 8 clusters, handling over 50PB of data.

Support for Network Function Virtualization

Up to this point, it might be considered a typical private cloud, regardless of its scale, but KDDI’s private cloud, MShip3, has undergone further unique tuning to run its Network Functions.

Tuning to ensure performance

The first is a hardware resource allocation design to enhance performance and responsiveness. For virtual machines used in Network Functions, CPU resources and memory are allocated exclusively to them, enabling them to deliver computing performance comparable to physical servers. This ensures high processing power and responsiveness during network congestion.

The second is the adoption of appropriate virtualization bypass technology to maximize performance. For network processing, which virtualization is not good at, we bypass the network processing by the Linux kernel and introduce a virtual switch, Open vSwitch, using DPDK (Data Plane Development Kit) by fully utilize CPU performance and achieve high network performance. We also enable the use of a technology called SR-IOV, which bypasses the any network software stacks for virtual machines.

In-House Operation and Development

Another significant feature is our development and operational structure.

While there are many “technicians who use the cloud,” those who utilize virtualization technologies such as OpenStack and “technicians who build the cloud” are rare. While it may sound a bit self-serving, the technical domains that comprise these private clouds require extremely broad technical skills and knowledge, both vertically (from hardware to the Linux kernel and the software layer providing APIs) and horizontally (from servers/computing to storage and networking). Therefore, we believe it is a very challenging field.

Therefore, KDDI maintains a high level of technical expertise by employing primarily internal engineers to build these clouds, and by collaborating with OSS support vendors and others. Furthermore, they adopt a full-cycle development style, handling not only development but also operations and planning and support for private clouds under the same team.

Summary

KDDI operates a large-scale private cloud, MShip3, in which it has full ownership. This article provided an overview of the scale of these private clouds and the technologies adopted.

In addition to the features mentioned above, KDDI’s private cloud also incorporates resource isolation design to improve availability, the use of virtual routers to achieve network flexibility, agility, and high performance, and unique cloud services and managed services tailored to KDDI’s specific internal needs. In future articles, we will introduce these key features of our private cloud in detail.