Fedora under construction?

Fedora’s quality makes complacency easy. But in truth, we’re always under construction — or we should be. You could call that constant disruption by different names. Risk positive. Forward leaning. Embracing change. Since inception, Fedora was intended to avoid the status quo. So what’s next for shaking up said status?

Season recap

As you’re probably aware, starting with Fedora 21 our releases are a bit different. We now have different editions, each serving a specific type of consumer. The editions today are Workstation, Cloud, and Server. The editions differ in mostly minor ways, though, and we build them mainly the same way. That’s why these editions aren’t the end goal of change. Rather, they’re a step in changing what Fedora releases.

Matthew Miller, Stephen Gallagher, and others have been steadily laying out a vision for change in Fedora. Admittedly, that vision’s high level. It uses words like “Rings” to simplify and amplify concepts that are about change. These concepts are well attuned to higher level Fedora goals. While editions can effectively broaden Fedora’s appeal to different consumer audiences, we also want to attract more makers of things.

These days, the largest maker audience is beyond just those who care about building a platform. As Red Hat’s Paul Cormier said last year, “The application is king.” Makers of applications and things above the OS are always on our minds. So how does this relate to the Fedora release of the future?

Season preview

For many years now, we’ve been building our release essentially the same way. We take a big bag of packages, like building blocks, and glue them together in a group that makes sense. We wrap that group in a shippable format (or several), sometimes with an installer, validate, and release it. Lately many Fedora folks are thinking about what that future release looks like, though. I would offer that the future release should be some combination of a strongly managed center, curated stacks, and an expanding nebula of containers.

Managed center? Does that mean the return of Fedora Core? No. Forget about Fedora Core. It’s dead, and it’s not coming back. Having the central part of the OS carefully managed in the community isn’t the same thing. Fortunately, there are emergent tools to do this. Among them, Project Atomic looks like one of the best bets in the space we care about. Atomic makes shipping an integrated, validated set of content easier. That content still comes from the packages we know well — kernel, glibc, bash, and others. But the rpm-ostree basis of Atomic can prevent slew in an installed system.

What do I mean by “slew”? Right now, the only Fedora release we know well is the one we put out at GA. After that, all bets are off. Any user potentially has a random subsets of updates on top. Saying “I’m using Fedora 22” is not very meaningful soon after release. That slew also makes validation, troubleshooting, diagnosis, and recovery unnecessarily hard. What if we manage this central content more carefully, using a model like rpm-ostree? We could validate a release more regularly, at least at a basic functional level.

There are other benefits as well. What if we constructed Rawhide using the rpm-ostree model? Imagine that an important core piece of the OS broke. We’d want to file that bug, track, fix, and update. Right now, hundreds of developers have to manually, sometimes painstakingly, fix their systems. This wastes large amounts of aggregate time. This problem is largely why many interested developers don’t run Rawhide as much as would be helpful. In the new model, if you’re invested in the problem, you can still work on it, of course. But the rest of the Rawhide users could, with a single command, back out to the previous tree and keep working. This keeps more interested (and interesting) people using Rawhide.

You could probably make a case for file system snapshots here. I think those are still useful for user data in this model. It’s not clear that snapshots would solve the slew problem above without imposing restrictions on them in some fashion. Would users be happy with that? Hard to say.

So what about these curated stacks? Well, to be honest it’s not yet clear what tech wins here. On one hand, enhancing rpm-ostree to allow layering might be a way forward.  Currently rpm-ostree is somewhat monolithic in that you can’t really mix or match stacks readily… yet. My understanding is the Atomic guys are thinking about this problem already, though, and how to solve it. So I expect the code will catch up to (and maybe overtake) the concept before we know it.

Alex Larsson is also doing interesting work on what he calls “sandboxed apps.” Sandboxing in this model might not be too different from containers. The concepts seem quite similar. Throw into this mix the recent progress on overlayfs, which is now part of the mainline Linux kernel. What you have is ripe ground for a new method to build and release big swaths of a platform.

Again, we have building blocks of a solution for interesting problems, some long-standing. The problems of the central core above are shared at common application layers as well. But it’s useful to detach them at key dividing lines for obvious reasons. In some way, shape, or form, it seems inevitable that Fedora must take a swing at these problems, even if it’s on a per-edition basis.

This leaves the nebula of containers I mentioned. A year and a half ago, containers were all the rage. People were thinking about how they affected the technology landscape for infrastructure. Perhaps some people reading this were thinking about containers as a fad. Time and consumers and commercial customers have pretty much proved that wrong. Containers allow app developers and users to move swiftly (or not) independent of the OS. The technology is here to stay, because it mostly makes OS maintenance issues someone else’s problem. In this case, ours — Fedora’s.

For a long time we’ve relied on people working the application layer to radically change their methods if they’re interested in deploying on Fedora. But frankly, time has shown the world doesn’t care to do that. So our choice is to adapt or face irrelevance. Matthew Miller has spoken to this point several times, so I won’t belabor it. My only other point is that, however we build the future Fedora release, it should make King App comfortable.

Renewal

For those people who might ask why we should take on this work, I’d start with a couple of thoughts. First, what’s in it for me as a contributor? Well, it depends first on whether you care about an edition that might use this model. I’m pretty involved in Workstation edition. For some time we’ve been interested in how to update Fedora for consumers at the OS and application level, rather than packages.  And something like a core + containers model using Atomic directly solves that problem. The Cloud SIG already has an Atomic image designed primarily for container management. Their user base has different expectations from Workstation. But clearly there’s room for great collaboration here, and I expect for Server too.

Another reason this is interesting is app vendor involvement in Fedora. Containers abstract the distribution problems away from application vendors. We know Fedora is in a lot of embedded hardware and other projects in the Real World. That problem space fits well with our current platform construction model. As I said before, we don’t necessarily need to stop doing that. But at the same time, embracing the rise of the app container lets us more effectively reach the developer audience. This is not just about our Workstation edition but, more generally, people who build and make interesting things. This is also somewhat tied up with the implementation of Rings. That’s why I look forward to Matthew and Stephen driving more detail and progress on that front.

Finally, by solving this problem we can effectively influence the value of future RHEL, as well as CentOS and our other family members. That value is one reason Red Hat invests time and resources in our community. Making changes to grow that value is always a win-win for Fedora as well.

The hero’s journey

So, does all this mean we won’t have live or installation ISOs in the future? Not necessarily. They could still be useful. It would depend, as most things do, on what it takes to validate and maintain all those things in the long run. For example, I don’t see this idea necessarily affecting spins. Communities around those releases are accustomed to how they build their bits.  But I think at least one (maybe more) of the official Fedora release editions will need to opt for this new model if we’re to make progress.

The entire set of changes needed isn’t yet clear, of course. One thing that is clear: release engineering process is, by definition, central. And there’s no doubt we’re looking at a healthy chunk of work. But I believe strongly enough in the possibilities that our team has an extra full-time person now to drive planning of those changes, consult with the Fedora release engineering team, and build a community around the tools needed. We are going to emphasize modularity, so the winning technology of tomorrow can plug in to releases down the road. The initial goal I’d like to set is to prototype a release of at least one Fedora edition for F23, and be part of F24 official release at GA.

That means it’s going to be an exciting year of construction ahead for Fedora. So please excuse our dust!