I’m extremely happy to announce that Randy Barlow has joined the Fedora Engineering team, effective today. Randy was previously a member of Red Hat’s team working on technologies like Pulp. You can find his fingerprints in many upstream repositories as a frequent contributor. This is fortunate for our team, since Randy will be contributing both to our applications infrastructure and to our release team. His experience with Pulp may come in handy too, since it may play a part in making next-generation Fedora content available to the public. I hope the Fedora community will join me in welcoming Randy!
My day job, as you probably know, is at Red Hat, where I manage the Fedora Engineering team. Our team provides engineering and design services for the Fedora Project, a collaborative community project which is the upstream source for a number of influential products. Not the least of these is Red Hat Enterprise Linux, of course.
We now have a job opening for a senior engineer in the USA to be part of our team.
About the job
Collaborating with the rest of the team, the community at large, and your colleagues in Red Hat, you’ll:
- Build tools and web services that will in turn build a more modular next-generation Fedora
- Do full-stack development, with a hand in everything from working with designers, to architecting and writing code, to deploying in test/stage/production, to maintenance and enhancement
- Automate, automate, automate all the things
- Co-create and develop our next community engagement system, Fedora Hubs
- Stay tuned into exciting work going on throughout Red Hat related to platform and other technologies
- As with all Fedora Engineering jobs, communicate openly and continually with the whole community, and build community around everything you do using open source best practices
About the Fedora Engineering team
Our team uses a lot of Python (flask and sqlalchemy currently rank highly), and we’ll be expanding outward in the future where it makes sense. We create code upstream that is widely consumable beyond just Fedora, and we deploy our work on both Red Hat Enterprise Linux and Fedora.
We do that work openly: collaboration via git repositories, rapid and constant communication via IRC, frequent discussion through our mailing lists, and gathering and building community around our work. Simply put, we love open.
Who we’re looking for
The right candidate is a team player, positive and constructive, fully engaged and passionately committed to delivering results with their colleagues, wherever they might be. This is a senior position, so we’re looking for someone who’s a proactive, detail-oriented self-starter, and has the ability to lead from the side on a few projects. We want you to be really good at Python and being Pythonic, and hopefully have some similar high-level language experience like Ruby or Java where you can help us grow, too.
Not interested in working in a Red Hat office? No sweat. We’re a distributed team, and this job is perfect for an experienced remote candidate. Remote doesn’t mean aloof, though. You need to be somewhat of a people person to crush this job, because good open source means collaboration. We meet up at least once a year as a team for Flock, but we work together constantly online.
I would be remiss in not pointing out this position back-fills some mighty big shoes. That’s why I’ve outlined some ideas for who we’re looking for in this post. That being said, you’ll also be working for a manager who’s a super guy. Just check the sidebar of this blog for proof. In all seriousness though, we all care about our teammates’ success, so you’re never on your own when you need a hand.
Enough already, where do I apply?
I don’t get as much of a chance these days to do things like patches or other technical contribution. But I had some time free the other day and used it to stick my hands directly into a cool project, Pagure (pronounced roughly “pag-yOOR,” or listen here). You may have read about Pagure in this Fedora Magazine article a few months back.
It was tremendously easy to get Pagure, fix a bug, test the fix, and contribute it back (see my pull request here). I specifically looked for “easyfix” bugs, since I knew I didn’t have a lot of time to spend to help. So I decided to work on this issue, a button showing up when it shouldn’t.
First I forked the repo in Pagure. Then I cloned my new fork, and set it up as documented in the README. In the clone, I checked out a new branch using the issue number as a name, issue839.
To track down the fix, I ran the app locally and duplicated the error. I looked at the CSS style containers for the button and some of the surrounding elements. Using that information, I did a text search on the code to find the file that was generating the button. Then I simply applied some logic to find the fix for the problem, even though I wasn’t really familiar with the code involved.
Thankfully Pagure has a test suite, so I could also check that my fix didn’t break any of the tests. Then I committed and pushed my changes, and made a pull request using the button in Pagure’s web page.
I also learned something useful. Since I forked the repo to do my fix and make a pull request, if I force-pushed changes using git to my branch from which I made the pull request, the pull request was automatically updated with the changes! This is probably expected by people who do this all the time, but since I’m new at it, I was excited.
Bugs like this are something that just about anyone with a small amount of beginning programming skill could fix. Pagure even has other bugs like this waiting for people to handle. Maybe one of them is waiting for you! 😉
I feel like my recent blog posts have all been about openings. But it’s nice to be able to offer them. Each internship on our job site is linked in the bullets below, so you can apply for any for which you feel qualified.
- Software engineering internship – This internship focuses primarily on Python software development. You’ll take designs and build them into widgets for our Fedora Hubs project. You’ll collaborate with our applications team on back-end technology for the community. You’ll also work with our infrastructure team on production deployment. This internship also includes other related projects, not limited to just Hubs. The intern is preferably in Westford, MA for the summer. But that’s not strictly required for a well-qualified candidate with experience working in a global open source project. (In other words, if you can succeed at keeping touch daily with the team as a remotee, don’t let the location throw you.)
- User experience and design internship – This intern will work directly with our designers Máirín Duffy and Ryan Lerch on research and design for Fedora projects such as Hubs. You’ll do user research and work on improving interaction in our web apps and other associated projects. You’ll collaborate with the other interns, our whole development team, and the Fedora community at large. This job is in Westford, MA.
- Web development internship – This intern will work primarily (but not exclusively) on web apps at the front end. You’ll turn design into reality, and streamline existing applications to designer specs to build a more unified project. As with our other internships, you’ll be able to explore and work on other projects too. You’ll work with our design team, the community Websites team and our application developers to create and improve web tools our contributors will use for years. This job is preferably in Westford, MA for the summer. But we’ll consider remote for an exceptionally qualified candidate.
These jobs should end up being quite competitive. So I encourage you to get your application in as soon as you can!
Last year, we were fortunate to have superb interns who produced fantastic results. Meghan Richardson worked on many of the initial designs for Hubs in its early stages, interviewing potential users and then creating the workflows and feel for this new tool. Nate Yazdani worked on statscache, which compiles data from our messaging bus to make it available quickly to other apps, without constantly crushing our data store.
Red Hat is a fantastic company at which to intern, and we pride ourselves on an open and inclusive culture. These are not “fetch coffee” jobs — you’ll work on real problems alongside real experts.
We can’t wait to see what you can do!
Here’s a fantastic opportunity in open source for folks from underrepresented backgrounds. The Fedora Engineering team has an Outreachy internship available December 2015 to March 2016. Applications are being accepted at the Outreachy site. The deadline is 1900 UTC Monday, November 2, 2015.
Are you familiar with Outreachy? It’s a program designed to boost participation in open source by underrepresented groups. My employer, Red Hat, is sponsoring this internship. The Fedora Project, which my team at Red Hat supports, is proud to partner in offering it. This Fedora internship is an opportunity to work directly with our senior software engineers on a major project called Fedora Hubs.
Hubs is an initiative to bring better and more timely information to Fedora’s large community of participants and contributors. Several of our engineers will collaborate with you throughout the internship period on this project. We’ll use Hubs as a foundation for regular, daily mentoring of our Outreachy intern. You’ll use best of breed technologies like Python, Bootstrap, jinja templates, and more. Best of all, you’ll work alongside the team in real time.
You can read more about the internship and qualifications here on our wiki. The page also has general information about Fedora, such as our mission and how we work together.
But this internship isn’t just extending your skills, or working with awesome people. (Even though that’s a great incentive!) It’s also about bringing the open source way to thousands of people. This project lets contributors build sub-communities, or hubs, around their interests. “Standing on the shoulders of giants” is one of the greatest benefits of open source. The same goes for using open source to build communities. It’s a natural fit and a wonderful outcome for a successful internship.
Are you interested in diving deeper into Hubs, to see what this internship is about? Check out this recent blog post from our own Máirín Duffy on the purpose, design, and mockups of the Hubs project.
Getting started at Flock
Like everyone on the Fedora Engineering team, I was in Rochester for the Flock conference last week. After several flight delays on our direct flight from DCA to Rochester, Justin Forbes, Ricky Elrod, and I finally arrived a little after 9:00pm — about four hours late. Thankfully Josh Boyer came to pick us up at the airport.
Flock had a team of organizers within OSAS (and Josh also assisted throughout). As a former FUDCon organizer, though, I know the value of extra hands showing up to do work. Since old habits die hard, I showed up expecting to help out behind the scenes. That means I didn’t get to see a huge amount of content I was personally interested in. But in return hopefully everyone had a smoother Flock experience, especially speakers.
When I arrived, I reported to Tom Callaway, Ruth Suehle, and Josh. They got the conference rooms opened, and I helped set up the speaker workstations. We worked pretty late, well after midnight. Things were looking a little bleak at that point, with execrable network bandwidth, no projectors, no screens, and no audio for the ballroom.
Fears and worries abate
Nevertheless, the next morning Josh and I got up early and grabbed coffee at nearby Tedward’s. This place was a godsend, although their 7:00am opening time forced us to walk around a bit until we could get in.
We went down to do some additional setup. The organizers had worked with Remy DeCausemaker to get a bunch of loaner projectors from RIT so we’d be ready for the first sessions at 10:00am. (EDIT: According to Remy, Tim Duffy and Dan Schneiderman are the heroes of this particular day; see comments below.) So at least our speakers would be in OK shape. I helped Josh and Tom get everything ready in those rooms, while Ruth made sure registration and other logistics were under control. I missed Matthew Miller’s keynote, but I’d seen at least some of the material previously.
After lunchtime, things continued to drastically improve. The rental projectors showed up, along with small screens for each room and big speakers for the ballroom. The wireless internet improved quite a bit when a switch flip occurred due to our conference starting up. (It was dismal Tuesday night!) We had all the speakers trained on how to record their talks locally, to get around the constrained network bandwidth.
Suddenly things were looking up! Not surprisingly, the Fedora Engineering team dinner that night at The Old Toad was much more enjoyable. Since I wasn’t overly worried about the conference experience for the speakers and attendees any longer, it was easier to relax and enjoy the company of the team. I was so happy that we were able to get together in one place, since we really only get to do that once a year. (Incidentally, our friend Stephen Smoogen was absent from Flock due to family commitments — we missed you, Smooge!)
I continued to monitor speaker rooms most of Wednesday and Thursday. I managed to make it to a couple sessions where I wasn’t sure there would be any senior Fedora leadership around. For example, I attended the Fedora Magazine session by Chris Roberts as well as most of the Fedora Hubs session by Máirín Duffy and Meghan Richardson.
I attended and loved Major Hayden‘s (of Rackspace fame) Thursday keynote on fighting impostor syndrome. It was one of the most practical that I’ve seen on this topic. I feel impostor syndrome is just a fancy way to refer to insecurity, a common trait for conscientious people. But that doesn’t make the strategies Major outlined any less useful or thoughtful. He gave a great talk — engaging and humorous without diluting the material. If you have a chance to invite him to a conference to speak, definitely do so!
I gave my own talk on Remote Ninjutsu on Thursday afternoon. The slides for the talk are here, although the video will be more useful for context. All the Flock 2015 videos are supposed to be available at some point in the next couple of weeks. Stay tuned for announcements about them.
The Thursday night social event at the Strong National Museum of Play was fantastic. It was a great way to blow off steam and enjoy the company of fellow Fedorans. I’m not sure how the organizers managed to find such a perfect venue!
Workshops and Flock wrap-up
On Friday I enjoyed the keynote by Jon Schull of eNable, the community that is flipping the script on prosthetics provision through 3D printing. It was a very moving look at how people are applying open source to make the world better for people in need.
Then the workshops beckoned. Now that I’d finished my Flock duties helping speakers and attendees, I was able to attend several sessions that were relevant to me personally, including the Fedora/CentOS rel-eng joint session, and my own on revamping the Flock software stack.
Once again, the Friday night social event at the George Eastman House was marvelous. It was a beautiful, grand mansion and the tour was quite interesting. I’d love to go back there sometime to see the exhibits I missed!
Flock conferences are always especially great for their hallway track. So many discussions can be had or progressed that way with high bandwidth. The challenge is always to move that discussion to a transparent context if it involves people not present, though. I’ve been seeing many trip reports from people’s blogs about Flock, and resulting list discussions, so I think that process is well underway.
Of course, that means Flock is a very engaging event. It takes a lot of attention and brainpower to shift focus for all those conversations! As a result, by Saturday afternoon I know I was fairly exhausted — in a good way, though. Several other people I know felt likewise, and commented on how well the conference had gone. In fact, I heard a number of comments that this was the best Flock, and even Fedora premier event, yet. The OSAS folks deserve special recognition for pulling off a fantastic conference.
Sunday started with a couple meetings, including with Matthew Miller and Jan Ku?ík, our new Fedora program manager. Then, after seeing a few other friends and colleagues off, I got to the airport. I relaxed in a lounge over beers with Kevin Fenzi, Jan Zeleny, and Stephen Tweedie, before we went to our respective flights. Then after a quick flight home, it was the usual “fun” making my way down I-95 from the airport to home. Monday morning was right around the corner…
Here’s to another great Flock, and to doing it again next year!
The team I manage at Red Hat, the Fedora Engineering team, includes people who work on Fedora system administration, release tooling, application development, and design. We have a job opening for an engineer to work with our infrastructure applications team on some challenging, fun, and forward-looking problems:
- Building and enhancing our tools for producing and shipping cloud images for a variety of providers (experience definitely required!)
- Working with the Fedora Cloud SIG and other community members to resolve issues related to the Fedora Cloud edition and associated tools and processes
- Collaborating with the rest of the Fedora team to automate, automate, automate All the Cloud Things
- Working with the rest of the team on non-Cloud related projects too, such as Fedora Hubs
- Staying abreast of and aligned with work going on throughout Red Hat related to Fedora and cloud technology
- As with all Fedora Engineering jobs, communicating openly and continually with the whole community, and building community around everything you do using open source best practices
Our team uses a lot of Python. We create code upstream that is widely consumable beyond just Fedora, and we deploy our work on both Red Hat Enterprise Linux and Fedora. We do that work openly: collaboration via git repositories, rapid and constant communication via IRC, frequent discussion through our mailing lists, and taking opportunities to gather and build community around our work. Simply put, we love open.
Although the description says the job is in Westford MA, USA, in reality we’re a highly distributed team. While this job is originally conceived as an entry- or journeyman-level engineer in the Westford (or possibly Raleigh NC) Red Hat office, we’re also open to experienced remotees outside the USA. The right candidate is a team player, fully engaged and passionately committed to delivering results with their colleagues, wherever they might be.
Does this sound interesting to you? Go read the full description of the job, and then apply online.
- Bad drivers at a lower layer (not PulseAudio’s problem)
- Bad hackery in the distro (ohai *buntu)
- People parroting years-old Internet “wisdom” (does that happen?)
But in a well-integrated system, PulseAudio is awesome. That’s definitely the case with Fedora Workstation. That includes Fedora 22, which is what I’m using now.
I’m attending a number of conference calls this week. Lots of them. The last two days have been eight hours of meetings. For example, today I’m at a Fedora Hubs workshop with fellow community members. I usually use my laptop to attend these conference calls. I use it with a USB headset attached. But having a headset on for that long is uncomfortable.
I also have a desktop computer in my office hooked to a big speaker set. So I use PulseAudio to route sound from my conference calls to those speakers. Now I don’t have to wear a headset to listen in.
I do this using the paprefs and pavucontrol utilities. I find these utilities very useful for more audio control. I can watch and move streams, all though an easy, comfortable GUI.
Thanks for the memory, PulseAudio
But that’s not all!
I was in the conference call, but there were updates available on the desktop computer. I decided to see what happened when I rebooted to get the updates. (Since my laptop was on the call, it wasn’t a problem.) I restarted the desktop, and the sound automatically moved back to the laptop speakers! That was cool — I didn’t lose any of the conference.
But it got better. After updates finished and the system restarted, I logged in. The sound automatically went back to the desktop speaker system! I was really impressed.
PulseAudio can track where audio streams are going, and send them back to the original devices when restarted. I love this feature! I remember the early days of Linux audio. It seemed like every single operation was hit-or-miss. I’m glad I don’t have to mess around with the system at that level anymore. I can just concentrate on my tasks, and things Just Work(tm).
Are you (1) coming to Flock 2015; (2) located, or planning to be, in the Boston, MA or Westford, MA area; and (3) interested in taking a free, power and wifi-equipped bus if offered?
If so, then pre-register by 1900 UTC Friday, June 5, and make sure you indicate in your registration you are located in the Boston/Westford area. I’ll collect the final number of these signups on Friday afternoon US/Eastern time. We’ll use that number to determine whether or not it’s economical to book the bus.
The announcement about the bus will be made next week, so there will be plenty of time to make appropriate travel plans either way.
Fedora’s quality makes complacency easy. But in truth, we’re always under construction — or we should be. You could call that constant disruption by different names. Risk positive. Forward leaning. Embracing change. Since inception, Fedora was intended to avoid the status quo. So what’s next for shaking up said status?
As you’re probably aware, starting with Fedora 21 our releases are a bit different. We now have different editions, each serving a specific type of consumer. The editions today are Workstation, Cloud, and Server. The editions differ in mostly minor ways, though, and we build them mainly the same way. That’s why these editions aren’t the end goal of change. Rather, they’re a step in changing what Fedora releases.
Matthew Miller, Stephen Gallagher, and others have been steadily laying out a vision for change in Fedora. Admittedly, that vision’s high level. It uses words like “Rings” to simplify and amplify concepts that are about change. These concepts are well attuned to higher level Fedora goals. While editions can effectively broaden Fedora’s appeal to different consumer audiences, we also want to attract more makers of things.
These days, the largest maker audience is beyond just those who care about building a platform. As Red Hat’s Paul Cormier said last year, “The application is king.” Makers of applications and things above the OS are always on our minds. So how does this relate to the Fedora release of the future?
For many years now, we’ve been building our release essentially the same way. We take a big bag of packages, like building blocks, and glue them together in a group that makes sense. We wrap that group in a shippable format (or several), sometimes with an installer, validate, and release it. Lately many Fedora folks are thinking about what that future release looks like, though. I would offer that the future release should be some combination of a strongly managed center, curated stacks, and an expanding nebula of containers.
Managed center? Does that mean the return of Fedora Core? No. Forget about Fedora Core. It’s dead, and it’s not coming back. Having the central part of the OS carefully managed in the community isn’t the same thing. Fortunately, there are emergent tools to do this. Among them, Project Atomic looks like one of the best bets in the space we care about. Atomic makes shipping an integrated, validated set of content easier. That content still comes from the packages we know well — kernel, glibc, bash, and others. But the rpm-ostree basis of Atomic can prevent slew in an installed system.
What do I mean by “slew”? Right now, the only Fedora release we know well is the one we put out at GA. After that, all bets are off. Any user potentially has a random subsets of updates on top. Saying “I’m using Fedora 22” is not very meaningful soon after release. That slew also makes validation, troubleshooting, diagnosis, and recovery unnecessarily hard. What if we manage this central content more carefully, using a model like rpm-ostree? We could validate a release more regularly, at least at a basic functional level.
There are other benefits as well. What if we constructed Rawhide using the rpm-ostree model? Imagine that an important core piece of the OS broke. We’d want to file that bug, track, fix, and update. Right now, hundreds of developers have to manually, sometimes painstakingly, fix their systems. This wastes large amounts of aggregate time. This problem is largely why many interested developers don’t run Rawhide as much as would be helpful. In the new model, if you’re invested in the problem, you can still work on it, of course. But the rest of the Rawhide users could, with a single command, back out to the previous tree and keep working. This keeps more interested (and interesting) people using Rawhide.
You could probably make a case for file system snapshots here. I think those are still useful for user data in this model. It’s not clear that snapshots would solve the slew problem above without imposing restrictions on them in some fashion. Would users be happy with that? Hard to say.
So what about these curated stacks? Well, to be honest it’s not yet clear what tech wins here. On one hand, enhancing rpm-ostree to allow layering might be a way forward. Currently rpm-ostree is somewhat monolithic in that you can’t really mix or match stacks readily… yet. My understanding is the Atomic guys are thinking about this problem already, though, and how to solve it. So I expect the code will catch up to (and maybe overtake) the concept before we know it.
Alex Larsson is also doing interesting work on what he calls “sandboxed apps.” Sandboxing in this model might not be too different from containers. The concepts seem quite similar. Throw into this mix the recent progress on overlayfs, which is now part of the mainline Linux kernel. What you have is ripe ground for a new method to build and release big swaths of a platform.
Again, we have building blocks of a solution for interesting problems, some long-standing. The problems of the central core above are shared at common application layers as well. But it’s useful to detach them at key dividing lines for obvious reasons. In some way, shape, or form, it seems inevitable that Fedora must take a swing at these problems, even if it’s on a per-edition basis.
This leaves the nebula of containers I mentioned. A year and a half ago, containers were all the rage. People were thinking about how they affected the technology landscape for infrastructure. Perhaps some people reading this were thinking about containers as a fad. Time and consumers and commercial customers have pretty much proved that wrong. Containers allow app developers and users to move swiftly (or not) independent of the OS. The technology is here to stay, because it mostly makes OS maintenance issues someone else’s problem. In this case, ours — Fedora’s.
For a long time we’ve relied on people working the application layer to radically change their methods if they’re interested in deploying on Fedora. But frankly, time has shown the world doesn’t care to do that. So our choice is to adapt or face irrelevance. Matthew Miller has spoken to this point several times, so I won’t belabor it. My only other point is that, however we build the future Fedora release, it should make King App comfortable.
For those people who might ask why we should take on this work, I’d start with a couple of thoughts. First, what’s in it for me as a contributor? Well, it depends first on whether you care about an edition that might use this model. I’m pretty involved in Workstation edition. For some time we’ve been interested in how to update Fedora for consumers at the OS and application level, rather than packages. And something like a core + containers model using Atomic directly solves that problem. The Cloud SIG already has an Atomic image designed primarily for container management. Their user base has different expectations from Workstation. But clearly there’s room for great collaboration here, and I expect for Server too.
Another reason this is interesting is app vendor involvement in Fedora. Containers abstract the distribution problems away from application vendors. We know Fedora is in a lot of embedded hardware and other projects in the Real World. That problem space fits well with our current platform construction model. As I said before, we don’t necessarily need to stop doing that. But at the same time, embracing the rise of the app container lets us more effectively reach the developer audience. This is not just about our Workstation edition but, more generally, people who build and make interesting things. This is also somewhat tied up with the implementation of Rings. That’s why I look forward to Matthew and Stephen driving more detail and progress on that front.
Finally, by solving this problem we can effectively influence the value of future RHEL, as well as CentOS and our other family members. That value is one reason Red Hat invests time and resources in our community. Making changes to grow that value is always a win-win for Fedora as well.
The hero’s journey
So, does all this mean we won’t have live or installation ISOs in the future? Not necessarily. They could still be useful. It would depend, as most things do, on what it takes to validate and maintain all those things in the long run. For example, I don’t see this idea necessarily affecting spins. Communities around those releases are accustomed to how they build their bits. But I think at least one (maybe more) of the official Fedora release editions will need to opt for this new model if we’re to make progress.
The entire set of changes needed isn’t yet clear, of course. One thing that is clear: release engineering process is, by definition, central. And there’s no doubt we’re looking at a healthy chunk of work. But I believe strongly enough in the possibilities that our team has an extra full-time person now to drive planning of those changes, consult with the Fedora release engineering team, and build a community around the tools needed. We are going to emphasize modularity, so the winning technology of tomorrow can plug in to releases down the road. The initial goal I’d like to set is to prototype a release of at least one Fedora edition for F23, and be part of F24 official release at GA.
That means it’s going to be an exciting year of construction ahead for Fedora. So please excuse our dust!