This week Red Hat announced its plan to put all its energy into CentOS Stream 8, resulting in the discontinuation of CentOS Linux 8 in one year’s time. CentOS Stream, originally announced in September of 2019, is a continuous release of RHEL which provides updates as soon as they are developed and verified. Many people who use CentOS Linux today now wonder if CentOS Stream 8 will be a suitable distribution for their use: is it tested, will it be stable? If you want to know what to expect from CentOS Stream, the best starting point is knowing how Red Hat Enterprise Linux is built. Let’s get into it!
Red Hat has been making Linux releases for such a long time, its original development methodology predates the agile manifesto. Historically, RHEL has been built behind closed doors, its plans held close enough that even the announcement of predictable 6-month minor / 3-year major releases seemed a monumental reveal during the RHEL 8 launch. Fortunately, how Red Hat makes Linux distributions has evolved, not just since calendar years started with “19”, and there have been multiple process generations since RHEL 8 launched just 18 months ago. While fundamentals like upstream first, copious quality engineering, ecosystem partnership, and customer care remain the same, we work continuously to improve how those fundamentals are implemented.
Let’s start with grounding: every RHEL minor release is based on the previous release, plus targeted backports of upstream development. Often, Red Hatters are the original authors of those patches, but there are no shortcuts: upstream acceptance is the first test every patch must go through before we start it through the journey that eventually leads to a patch’s integration in the release. Even then, this is about an upstream patch existing, but that alone will not guarantee a patch’s inclusion.
Any decision to introduce an upstream change into RHEL is a team decision and the team is large: developers, quality engineers, support personnel, product owners, and various partners all work together on priority and feasibility. Once a decision is reached and commitments are made, only then do developers and quality engineers begin writing code. As you probably know, in the most congenial of rivalries, developers try to write code that nobody can break and quality engineers create batteries of ways to break the code developers write. This brings us to the second key place where Red Hat invests: tests.
We write tests for everything: unit tests, systemic tests, kernel and userspace ABI conformance tests, performance tests, dependency tests, architecture tests, driver tests, load tests, and many more. Having tests is foundational, but it is their application that brings meaning. This brings us to the third key area where Red Hat invests: process infrastructure.
For the last several years, Red Hat has worked on a series of “Always Ready” operating system initiatives. The goal is as simple as the name suggests: at any moment in time the OS is ready for release. It’s easier to describe than it is to implement. In complex systems, so many things can have unintended consequences. To handle this we use layers of automation, incrementally building confidence in changes, before they are integrated and released into the distribution. Here is a high-level sketch of the process every single change in RHEL must go through to be included:
When a change is targeted at RHEL, multiple incremental steps occur before it is actually included. Changes are built, but the only certain outcome is that a CI system will run a suite of tests on the builds (the build is not yet made available for general use). If those tests pass, a second round of preverification specific to the code change occurs (not yet good enough). And if those tests pass, the change is tentatively included in the errata system and subject to further verification (it’s still not ready to publish). Systemic test suites run on the combined whole to verify the gestalt functionality. And if those tests pass, the build will finally make it into the space where CentOS Stream systems recognize it as an available update. It’s a long pipeline and many changes move through it every single day. For those interested in more of the vision and architecture of this system, you can read more in CentOS Stream is Continuous Delivery!
While the description of this system may seem elegant and reassuring, watching it in action can feel quite the opposite: The more testing is done the more bugs are found- and Red Hat does a whole lot of testing. Historically, RHEL development has been done behind closed doors, isolating people from the routine bug identification and remediation process, only allowing the world to see the end result. In the future, as RHEL development becomes more transparent, as we approach RHEL 9, this process will become uncomfortably visible. While the testing systems are built to prevent such failures from reaching end users, anybody who wants to look deeper may be surprised at how messy operating system development can be!
Finally, for those who wonder how soon all this will map to CentOS Stream, we have good news: it is already happening today with RHEL 8.4 and CentOS Stream 8! At the same time these RHEL builds are verified, they are also delivered to CentOS Stream. Of course we aren’t done yet: CentOS Stream has not yet realized its mission of adding a developer community around RHEL, that is where we are headed, into a place where there are more options to engage with Red Hat and shape future RHEL. There is always room for improvement, from better tests to more facets in future collaboration, we are excited to share building RHEL with you so that we build a better OS with and for you.