Category Archives: Linux

The O/S that keeps me free (as in speech and beer)

Flock 2015 thoughts.

Getting started at Flock

Like everyone on the Fedora Engineering team, I was in Rochester for the Flock conference last week. After several flight delays on our direct flight from DCA to Rochester, Justin Forbes, Ricky Elrod, and I finally arrived a little after 9:00pm — about four hours late. Thankfully Josh Boyer came to pick us up at the airport.

Flock had a team of organizers within OSAS (and Josh also assisted throughout). As a former FUDCon organizer, though, I know the value of extra hands showing up to do work. Since old habits die hard, I showed up expecting to help out behind the scenes. That means I didn’t get to see a huge amount of content I was personally interested in. But in return hopefully everyone had a smoother Flock experience, especially speakers.

When I arrived, I reported to Tom Callaway, Ruth Suehle, and Josh. They got the conference rooms opened, and I helped set up the speaker workstations. We worked pretty late, well after midnight. Things were looking a little bleak at that point, with execrable network bandwidth, no projectors, no screens, and no audio for the ballroom.

Fears and worries abate

Nevertheless, the next morning Josh and I got up early and grabbed coffee at nearby Tedward’s. This place was a godsend, although their 7:00am opening time forced us to walk around a bit until we could get in.

We went down to do some additional setup. The organizers had worked with Remy DeCausemaker to get a bunch of loaner projectors from RIT so we’d be ready for the first sessions at 10:00am. (EDIT: According to Remy, Tim Duffy and Dan Schneiderman are the heroes of this particular day; see comments below.) So at least our speakers would be in OK shape. I helped Josh and Tom get everything ready in those rooms, while Ruth made sure registration and other logistics were under control. I missed Matthew Miller’s keynote, but I’d seen at least some of the material previously.

After lunchtime, things continued to drastically improve. The rental projectors showed up, along with small screens for each room and big speakers for the ballroom. The wireless internet improved quite a bit when a switch flip occurred due to our conference starting up. (It was dismal Tuesday night!) We had all the speakers trained on how to record their talks locally, to get around the constrained network bandwidth.

Suddenly things were looking up! Not surprisingly, the Fedora Engineering team dinner that night at The Old Toad was much more enjoyable. Since I wasn’t overly worried about the conference experience for the speakers and attendees any longer, it was easier to relax and enjoy the company of the team. I was so happy that we were able to get together in one place, since we really only get to do that once a year. (Incidentally, our friend Stephen Smoogen was absent from Flock due to family commitments — we missed you, Smooge!)

Fedora contributors at Flock gather at Victoire for dinner
Fedora contributors at Flock gather at Victoire for dinner

I continued to monitor speaker rooms most of Wednesday and Thursday. I managed to make it to a couple sessions where I wasn’t sure there would be any senior Fedora leadership around. For example, I attended the Fedora Magazine session by Chris Roberts as well as most of the Fedora Hubs session by Máirín Duffy and Meghan Richardson.

I attended and loved Major Hayden‘s (of Rackspace fame) Thursday keynote on fighting impostor syndrome. It was one of the most practical that I’ve seen on this topic. I feel impostor syndrome is just a fancy way to refer to insecurity, a common trait for conscientious people. But that doesn’t make the strategies Major outlined any less useful or thoughtful. He gave a great talk — engaging and humorous without diluting the material. If you have a chance to invite him to a conference to speak, definitely do so!

I gave my own talk on Remote Ninjutsu on Thursday afternoon. The slides for the talk are here, although the video will be more useful for context. All the Flock 2015 videos are supposed to be available at some point in the next couple of weeks. Stay tuned for announcements about them.

The Thursday night social event at the Strong National Museum of Play was fantastic. It was a great way to blow off steam and enjoy the company of fellow Fedorans. I’m not sure how the organizers managed to find such a perfect venue!

Workshops and Flock wrap-up

On Friday I enjoyed the keynote by Jon Schull of eNable, the community that is flipping the script on prosthetics provision through 3D printing. It was a very moving look at how people are applying open source to make the world better for people in need.

Then the workshops beckoned. Now that I’d finished my Flock duties helping speakers and attendees, I was able to attend several sessions that were relevant to me personally, including the Fedora/CentOS rel-eng joint session, and my own on revamping the Flock software stack.

Once again, the Friday night social event at the George Eastman House was marvelous. It was a beautiful, grand mansion and the tour was quite interesting. I’d love to go back there sometime to see the exhibits I missed!

The music parlor in the George Eastman House
The music parlor in the George Eastman House

Flock conferences are always especially great for their hallway track. So many discussions can be had or progressed that way with high bandwidth. The challenge is always to move that discussion to a transparent context if it involves people not present, though. I’ve been seeing many trip reports from people’s blogs about Flock, and resulting list discussions, so I think that process is well underway.

Of course, that means Flock is a very engaging event. It takes a lot of attention and brainpower to shift focus for all those conversations! As a result, by Saturday afternoon I know I was fairly exhausted — in a good way, though. Several other people I know felt likewise, and commented on how well the conference had gone. In fact, I heard a number of comments that this was the best Flock, and even Fedora premier event, yet. The OSAS folks deserve special recognition for pulling off a fantastic conference.

Sunday started with a couple meetings, including with Matthew Miller and Jan Ku?ík, our new Fedora program manager. Then, after seeing a few other friends and colleagues off, I got to the airport. I relaxed in a lounge over beers with Kevin Fenzi, Jan Zeleny, and Stephen Tweedie, before we went to our respective flights. Then after a quick flight home, it was the usual “fun” making my way down I-95 from the airport to home. Monday morning was right around the corner…

Here’s to another great Flock, and to doing it again next year!

Flock attendees wind down after the conference ends... with more hacking!
Flock attendees wind down after the conference ends… with more hacking!

Fedora Engineering team opening.

The team I manage at Red Hat, the Fedora Engineering team, includes people who work on Fedora system administration, release tooling, application development, and design. We have a job opening for an engineer to work with our infrastructure applications team on some challenging, fun, and forward-looking problems:

  • Building and enhancing our tools for producing and shipping cloud images for a variety of providers (experience definitely required!)
  • Working with the Fedora Cloud SIG and other community members to resolve issues related to the Fedora Cloud edition and associated tools and processes
  • Collaborating with the rest of the Fedora team to automate, automate, automate All the Cloud Things
  • Working with the rest of the team on non-Cloud related projects too, such as Fedora Hubs
  • Staying abreast of and aligned with work going on throughout Red Hat related to Fedora and cloud technology
  • As with all Fedora Engineering jobs, communicating openly and continually with the whole community, and building community around everything you do using open source best practices

Our team uses a lot of Python. We create code upstream that is widely consumable beyond just Fedora, and we deploy our work on both Red Hat Enterprise Linux and Fedora. We do that work openly: collaboration via git repositories, rapid and constant communication via IRC, frequent discussion through our mailing lists, and taking opportunities to gather and build community around our work. Simply put, we love open.

Although the description says the job is in Westford MA, USA, in reality we’re a highly distributed team. While this job is originally conceived as an entry- or journeyman-level engineer in the Westford (or possibly Raleigh NC) Red Hat office, we’re also open to experienced remotees outside the USA. The right candidate is a team player, fully engaged and passionately committed to delivering results with their colleagues, wherever they might be.

Does this sound interesting to you? Go read the full description of the job, and then apply online.

PulseAudio is still awesome.

PulseAudio is an indispensable part of my Fedora Workstation. While PulseAudio may have detractors, my experience has been all good. Most sound problems I’ve seen come down to one of the following:

  • Bad drivers at a lower layer (not PulseAudio’s problem)
  • Bad hackery in the distro (ohai *buntu)
  • People parroting years-old Internet “wisdom” (does that happen?)

But in a well-integrated system, PulseAudio is awesome. That’s definitely the case with Fedora Workstation. That includes Fedora 22, which is what I’m using now.

Meetingpalooza

I’m attending a number of conference calls this week. Lots of them. The last two days have been eight hours of meetings. For example, today I’m at a Fedora Hubs workshop with fellow community members. I usually use my laptop to attend these conference calls. I use it with a USB headset attached. But having a headset on for that long is uncomfortable.

I also have a desktop computer in my office hooked to a big speaker set. So I use PulseAudio to route sound from my conference calls to those speakers. Now I don’t have to wear a headset to listen in.

I do this using the paprefs and pavucontrol utilities. I find these utilities very useful for more audio control. I can watch and move streams, all though an easy, comfortable GUI.

My home office setup, courtesy of PulseAudio.
My home office setup, courtesy of PulseAudio.

Thanks for the memory, PulseAudio

But that’s not all!

I was in the conference call, but there were updates available on the desktop computer. I decided to see what happened when I rebooted to get the updates. (Since my laptop was on the call, it wasn’t a problem.) I restarted the desktop, and the sound automatically moved back to the laptop speakers! That was cool — I didn’t lose any of the conference.

But it got better. After updates finished and the system restarted, I logged in. The sound automatically went back to the desktop speaker system! I was really impressed.

PulseAudio can track where audio streams are going, and send them back to the original devices when restarted. I love this feature! I remember the early days of Linux audio. It seemed like every single operation was hit-or-miss. I’m glad I don’t have to mess around with the system at that level anymore. I can just concentrate on my tasks, and things Just Work(tm).

Flock bus from Boston/Westford.

Are you (1) coming to Flock 2015; (2) located, or planning to be, in the Boston, MA or Westford, MA area; and (3) interested in taking a free, power and wifi-equipped bus if offered?

If so, then pre-register by 1900 UTC Friday, June 5, and make sure you indicate in your registration you are located in the Boston/Westford area. I’ll collect the final number of these signups on Friday afternoon US/Eastern time. We’ll use that number to determine whether or not it’s economical to book the bus.

The announcement about the bus will be made next week, so there will be plenty of time to make appropriate travel plans either way.

Fedora under construction?

Fedora’s quality makes complacency easy. But in truth, we’re always under construction — or we should be. You could call that constant disruption by different names. Risk positive. Forward leaning. Embracing change. Since inception, Fedora was intended to avoid the status quo. So what’s next for shaking up said status?

2178390257_66010d4b08_o
Courtesy of US Library of Congress.

Season recap

As you’re probably aware, starting with Fedora 21 our releases are a bit different. We now have different editions, each serving a specific type of consumer. The editions today are Workstation, Cloud, and Server. The editions differ in mostly minor ways, though, and we build them mainly the same way. That’s why these editions aren’t the end goal of change. Rather, they’re a step in changing what Fedora releases.

Matthew Miller, Stephen Gallagher, and others have been steadily laying out a vision for change in Fedora. Admittedly, that vision’s high level. It uses words like “Rings” to simplify and amplify concepts that are about change. These concepts are well attuned to higher level Fedora goals. While editions can effectively broaden Fedora’s appeal to different consumer audiences, we also want to attract more makers of things.

These days, the largest maker audience is beyond just those who care about building a platform. As Red Hat’s Paul Cormier said last year, “The application is king.” Makers of applications and things above the OS are always on our minds. So how does this relate to the Fedora release of the future?

Season preview

For many years now, we’ve been building our release essentially the same way. We take a big bag of packages, like building blocks, and glue them together in a group that makes sense. We wrap that group in a shippable format (or several), sometimes with an installer, validate, and release it. Lately many Fedora folks are thinking about what that future release looks like, though. I would offer that the future release should be some combination of a strongly managed center, curated stacks, and an expanding nebula of containers.

Managed center? Does that mean the return of Fedora Core? No. Forget about Fedora Core. It’s dead, and it’s not coming back. Having the central part of the OS carefully managed in the community isn’t the same thing. Fortunately, there are emergent tools to do this. Among them, Project Atomic looks like one of the best bets in the space we care about. Atomic makes shipping an integrated, validated set of content easier. That content still comes from the packages we know well — kernel, glibc, bash, and others. But the rpm-ostree basis of Atomic can prevent slew in an installed system.

What do I mean by “slew”? Right now, the only Fedora release we know well is the one we put out at GA. After that, all bets are off. Any user potentially has a random subsets of updates on top. Saying “I’m using Fedora 22” is not very meaningful soon after release. That slew also makes validation, troubleshooting, diagnosis, and recovery unnecessarily hard. What if we manage this central content more carefully, using a model like rpm-ostree? We could validate a release more regularly, at least at a basic functional level.

There are other benefits as well. What if we constructed Rawhide using the rpm-ostree model? Imagine that an important core piece of the OS broke. We’d want to file that bug, track, fix, and update. Right now, hundreds of developers have to manually, sometimes painstakingly, fix their systems. This wastes large amounts of aggregate time. This problem is largely why many interested developers don’t run Rawhide as much as would be helpful. In the new model, if you’re invested in the problem, you can still work on it, of course. But the rest of the Rawhide users could, with a single command, back out to the previous tree and keep working. This keeps more interested (and interesting) people using Rawhide.

You could probably make a case for file system snapshots here. I think those are still useful for user data in this model. It’s not clear that snapshots would solve the slew problem above without imposing restrictions on them in some fashion. Would users be happy with that? Hard to say.

So what about these curated stacks? Well, to be honest it’s not yet clear what tech wins here. On one hand, enhancing rpm-ostree to allow layering might be a way forward.  Currently rpm-ostree is somewhat monolithic in that you can’t really mix or match stacks readily… yet. My understanding is the Atomic guys are thinking about this problem already, though, and how to solve it. So I expect the code will catch up to (and maybe overtake) the concept before we know it.

Alex Larsson is also doing interesting work on what he calls “sandboxed apps.” Sandboxing in this model might not be too different from containers. The concepts seem quite similar. Throw into this mix the recent progress on overlayfs, which is now part of the mainline Linux kernel. What you have is ripe ground for a new method to build and release big swaths of a platform.

Again, we have building blocks of a solution for interesting problems, some long-standing. The problems of the central core above are shared at common application layers as well. But it’s useful to detach them at key dividing lines for obvious reasons. In some way, shape, or form, it seems inevitable that Fedora must take a swing at these problems, even if it’s on a per-edition basis.

10921733615_56f0835096_o
Courtesy of Glyn Lowe (CC-BY 2.0).

This leaves the nebula of containers I mentioned. A year and a half ago, containers were all the rage. People were thinking about how they affected the technology landscape for infrastructure. Perhaps some people reading this were thinking about containers as a fad. Time and consumers and commercial customers have pretty much proved that wrong. Containers allow app developers and users to move swiftly (or not) independent of the OS. The technology is here to stay, because it mostly makes OS maintenance issues someone else’s problem. In this case, ours — Fedora’s.

For a long time we’ve relied on people working the application layer to radically change their methods if they’re interested in deploying on Fedora. But frankly, time has shown the world doesn’t care to do that. So our choice is to adapt or face irrelevance. Matthew Miller has spoken to this point several times, so I won’t belabor it. My only other point is that, however we build the future Fedora release, it should make King App comfortable.

Renewal

For those people who might ask why we should take on this work, I’d start with a couple of thoughts. First, what’s in it for me as a contributor? Well, it depends first on whether you care about an edition that might use this model. I’m pretty involved in Workstation edition. For some time we’ve been interested in how to update Fedora for consumers at the OS and application level, rather than packages.  And something like a core + containers model using Atomic directly solves that problem. The Cloud SIG already has an Atomic image designed primarily for container management. Their user base has different expectations from Workstation. But clearly there’s room for great collaboration here, and I expect for Server too.

Another reason this is interesting is app vendor involvement in Fedora. Containers abstract the distribution problems away from application vendors. We know Fedora is in a lot of embedded hardware and other projects in the Real World. That problem space fits well with our current platform construction model. As I said before, we don’t necessarily need to stop doing that. But at the same time, embracing the rise of the app container lets us more effectively reach the developer audience. This is not just about our Workstation edition but, more generally, people who build and make interesting things. This is also somewhat tied up with the implementation of Rings. That’s why I look forward to Matthew and Stephen driving more detail and progress on that front.

Finally, by solving this problem we can effectively influence the value of future RHEL, as well as CentOS and our other family members. That value is one reason Red Hat invests time and resources in our community. Making changes to grow that value is always a win-win for Fedora as well.

The hero’s journey

So, does all this mean we won’t have live or installation ISOs in the future? Not necessarily. They could still be useful. It would depend, as most things do, on what it takes to validate and maintain all those things in the long run. For example, I don’t see this idea necessarily affecting spins. Communities around those releases are accustomed to how they build their bits.  But I think at least one (maybe more) of the official Fedora release editions will need to opt for this new model if we’re to make progress.

The entire set of changes needed isn’t yet clear, of course. One thing that is clear: release engineering process is, by definition, central. And there’s no doubt we’re looking at a healthy chunk of work. But I believe strongly enough in the possibilities that our team has an extra full-time person now to drive planning of those changes, consult with the Fedora release engineering team, and build a community around the tools needed. We are going to emphasize modularity, so the winning technology of tomorrow can plug in to releases down the road. The initial goal I’d like to set is to prototype a release of at least one Fedora edition for F23, and be part of F24 official release at GA.

That means it’s going to be an exciting year of construction ahead for Fedora. So please excuse our dust!

2363688612_6d71737477_b
Courtesy of Jakob Montrasio (CC-BY).

 

Another new Fedora team member.

I’m excited to tell you about another* new addition to the Fedora Engineering team. In mid-April, Laura Abbott joins our kernel team. She comes to us most recently from Qualcomm. Her projects there included memory management and debugging support. She’s also a regular contributor to the Linux kernel upstream. Laura will work with Josh Boyer and Justin Forbes on projects relating to both the upstream and Fedora kernel.  We’re happy to have Laura joining our team soon. Please help us extend her a warm welcome.

* As you know from my last announcement, Adam Miller also joins us in April. He’ll work on release tools and process for Project Atomic and other leading edge technologies.

New addition.

I’m proud and pleased to announce that Adam Miller is joining the Fedora Engineering team in mid-April, as our new release infrastructure guru. Adam is not new to our community — he’s been involved in Fedora for a number of years. He also has a long history of working on automation, release, and continuous integration in OpenShift, and I’m thrilled he’ll be bringing his experience and energy to our team. Among other things, Adam will be working on the next generation of release tools and deliverables in Fedora, in conjunction with the release engineering team and our community. I hope the Fedora community will join me in welcoming him.

New features in Fedora 22 Workstation.

Matthias Clasen recently posted some updates on the Fedora development list about new features in Fedora 22 Workstation. As you may know, we’re getting ready to issue an Alpha, so it’s a great time to try out these changes.

  • Notifications have improved significantly in Fedora 22. The message tray no longer hides in the bottom, but is subsumed into the top bar. You can review them from the calendar popup. (Here are the original mockups.) Please try out the notifications with your favorite applications and report bugs if you find functional problems.
  • The GNOME Shell theme has been refreshed.
  • The login screen in Fedora 22 Workstation now runs over Wayland, but falls back to the standard Xorg server if something goes wrong. This is part of Fedora’s overall strategy of getting things running on Wayland early. Wayland as a session option was available in Fedora 21, but we are ultimately moving toward having it as the default in a future release. Note that the fallback calls Xorg in a slightly different way, so you may see bugs (and filing a bug report is welcome).
  • The UI for installation of codecs, fonts, MIME types, and so on is now handled by GNOME Software. This change further enhances the sense of GNOME Software as a one-stop shop for installing things, which was a highly praised feature in Fedora 21.
  • Nautilus has had multiple improvements, including more pleasant and logical context menus and controls (here are the designs).

Fedora 22 Alpha is scheduled to release on March 10. We hope you’ll give it a try, so you can test drive these new features. Those of you knowledgeable about Fedora 22 pre-releases can find early test candidate images for Alpha in the usual places. Please use Bugzilla to file helpful bug reports. If you need help filing a bug report, this wiki page may be useful.

Fedora kernel engineer.

Red Hat has an immediate opening for a full-time engineer to join the kernel team in Fedora Engineering. This job will work with Josh Boyer and Justin Forbes to maintain and improve the kernel in Fedora, and participate and contribute to upstream development and testing. This job interacts with the Fedora team and community, the RHEL kernel engineering groups, and the upstream kernel community.

Ideally we’re looking for someone with significant kernel experience. We want someone who can help manage our own kernel releases. But also we want to get patches and code upstream where it can benefit the entire community, for example to improve hardware support. Red Hat’s a challenging and exciting place to work, and the Fedora team is a great bunch of people to work with. A talented, motivated kernel hacker will find plenty of opportunity here for growth and collaboration.

Applications are being accepted now. You can find more information about the job, and apply online, via the posting here on our Red Hat Jobs site.

 

Design Team FAD 2015.

I’m here in Westford over the weekend with other members of the Fedora Design Team for their Fedora activity day. I traveled up to chilly Boston on Thursday midday, so I could assist by transporting people from the airport to our hotel in Westford.

On Friday, we convened in the Red Hat office in Westford. As is usual for my Westford visits, I pretty much spent the day running from place to place taking care of things unrelated to the Design team, but which needed to be done in my role as a manager. Fortunately most of the day was spent by the team figuring out policy and processes. I don’t feel like my input was needed there, or even appropriate since I’m not a frequent contributor to design tasks — as much as I love the team! So it all worked out for the best, I think.

Today, though, I was able to contribute. One of the major task areas was to do issue triage and fix up the team’s Trac instance. That was something I was (somewhat) qualified to do. I helped the team go through all the pending tickets, closing stale tickets (no response, unclear goals, redundant, etc.).

Then, using the categories developed by the team on Friday, I helped update the parameters on the Trac instance to match. I also set up a bunch of reports on Trac to match relevant agenda items for the reboot of the Design team meetings. This way, they can call up a set of reports to follow up on tickets methodically.

I even learned some good SQL-fu thanks to my buddy Langdon White. I finally got a start shedding my misunderstanding of joins, so I can do more complicated queries. One of the results was this report, which tells the team when an issue reporter is not responsive to questions, in line with the processes the team worked up on Friday.

People tried hard to make sure the FAD was remote accessible, so if you couldn’t be here, you could still monitor or participate. It was difficult to keep some of the facilities working. For instance, this afternoon I discovered that someone had bounced us from our own room on OpenTokRTC. That made it, well, rather difficult for us to broadcast there. I hope remote attendees will understand the difficulties and be confident we tried to make this a decent remote event.

Tonight I’ll probably think about some additional reports, and then do a little personal work. Tomorrow will be some more work sprints until I start chauffeuring people to the airport after lunch. I was happy to be part of the group and to help the participants have an effective FAD.