I don’t get as much of a chance these days to do things like patches or other technical contribution. But I had some time free the other day and used it to stick my hands directly into a cool project, Pagure (pronounced roughly “pag-yOOR,” or listen here). You may have read about Pagure in this Fedora Magazine article a few months back.
It was tremendously easy to get Pagure, fix a bug, test the fix, and contribute it back (see my pull request here). I specifically looked for “easyfix” bugs, since I knew I didn’t have a lot of time to spend to help. So I decided to work on this issue, a button showing up when it shouldn’t.
First I forked the repo in Pagure. Then I cloned my new fork, and set it up as documented in the README. In the clone, I checked out a new branch using the issue number as a name, issue839.
To track down the fix, I ran the app locally and duplicated the error. I looked at the CSS style containers for the button and some of the surrounding elements. Using that information, I did a text search on the code to find the file that was generating the button. Then I simply applied some logic to find the fix for the problem, even though I wasn’t really familiar with the code involved.
Thankfully Pagure has a test suite, so I could also check that my fix didn’t break any of the tests. Then I committed and pushed my changes, and made a pull request using the button in Pagure’s web page.
I also learned something useful. Since I forked the repo to do my fix and make a pull request, if I force-pushed changes using git to my branch from which I made the pull request, the pull request was automatically updated with the changes! This is probably expected by people who do this all the time, but since I’m new at it, I was excited.
Bugs like this are something that just about anyone with a small amount of beginning programming skill could fix. Pagure even has other bugs like this waiting for people to handle. Maybe one of them is waiting for you! 😉
Flock had a team of organizers within OSAS (and Josh also assisted throughout). As a former FUDCon organizer, though, I know the value of extra hands showing up to do work. Since old habits die hard, I showed up expecting to help out behind the scenes. That means I didn’t get to see a huge amount of content I was personally interested in. But in return hopefully everyone had a smoother Flock experience, especially speakers.
When I arrived, I reported to Tom Callaway, Ruth Suehle, and Josh. They got the conference rooms opened, and I helped set up the speaker workstations. We worked pretty late, well after midnight. Things were looking a little bleak at that point, with execrable network bandwidth, no projectors, no screens, and no audio for the ballroom.
Fears and worries abate
Nevertheless, the next morning Josh and I got up early and grabbed coffee at nearby Tedward’s. This place was a godsend, although their 7:00am opening time forced us to walk around a bit until we could get in.
We went down to do some additional setup. The organizers had worked with Remy DeCausemaker to get a bunch of loaner projectors from RIT so we’d be ready for the first sessions at 10:00am. (EDIT: According to Remy, Tim Duffy and Dan Schneiderman are the heroes of this particular day; see comments below.) So at least our speakers would be in OK shape. I helped Josh and Tom get everything ready in those rooms, while Ruth made sure registration and other logistics were under control. I missed Matthew Miller’s keynote, but I’d seen at least some of the material previously.
After lunchtime, things continued to drastically improve. The rental projectors showed up, along with small screens for each room and big speakers for the ballroom. The wireless internet improved quite a bit when a switch flip occurred due to our conference starting up. (It was dismal Tuesday night!) We had all the speakers trained on how to record their talks locally, to get around the constrained network bandwidth.
Suddenly things were looking up! Not surprisingly, the Fedora Engineering team dinner that night at The Old Toad was much more enjoyable. Since I wasn’t overly worried about the conference experience for the speakers and attendees any longer, it was easier to relax and enjoy the company of the team. I was so happy that we were able to get together in one place, since we really only get to do that once a year. (Incidentally, our friend Stephen Smoogen was absent from Flock due to family commitments — we missed you, Smooge!)
I continued to monitor speaker rooms most of Wednesday and Thursday. I managed to make it to a couple sessions where I wasn’t sure there would be any senior Fedora leadership around. For example, I attended the Fedora Magazine session by Chris Roberts as well as most of the Fedora Hubs session by Máirín Duffy and Meghan Richardson.
I attended and loved Major Hayden‘s (of Rackspace fame) Thursday keynote on fighting impostor syndrome. It was one of the most practical that I’ve seen on this topic. I feel impostor syndrome is just a fancy way to refer to insecurity, a common trait for conscientious people. But that doesn’t make the strategies Major outlined any less useful or thoughtful. He gave a great talk — engaging and humorous without diluting the material. If you have a chance to invite him to a conference to speak, definitely do so!
I gave my own talk on Remote Ninjutsu on Thursday afternoon. The slides for the talk are here, although the video will be more useful for context. All the Flock 2015 videos are supposed to be available at some point in the next couple of weeks. Stay tuned for announcements about them.
The Thursday night social event at the Strong National Museum of Play was fantastic. It was a great way to blow off steam and enjoy the company of fellow Fedorans. I’m not sure how the organizers managed to find such a perfect venue!
Workshops and Flock wrap-up
On Friday I enjoyed the keynote by Jon Schull of eNable, the community that is flipping the script on prosthetics provision through 3D printing. It was a very moving look at how people are applying open source to make the world better for people in need.
Then the workshops beckoned. Now that I’d finished my Flock duties helping speakers and attendees, I was able to attend several sessions that were relevant to me personally, including the Fedora/CentOS rel-eng joint session, and my own on revamping the Flock software stack.
Once again, the Friday night social event at the George Eastman House was marvelous. It was a beautiful, grand mansion and the tour was quite interesting. I’d love to go back there sometime to see the exhibits I missed!
Flock conferences are always especially great for their hallway track. So many discussions can be had or progressed that way with high bandwidth. The challenge is always to move that discussion to a transparent context if it involves people not present, though. I’ve been seeing many trip reports from people’s blogs about Flock, and resulting list discussions, so I think that process is well underway.
Of course, that means Flock is a very engaging event. It takes a lot of attention and brainpower to shift focus for all those conversations! As a result, by Saturday afternoon I know I was fairly exhausted — in a good way, though. Several other people I know felt likewise, and commented on how well the conference had gone. In fact, I heard a number of comments that this was the best Flock, and even Fedora premier event, yet. The OSAS folks deserve special recognition for pulling off a fantastic conference.
Sunday started with a couple meetings, including with Matthew Miller and Jan Ku?ík, our new Fedora program manager. Then, after seeing a few other friends and colleagues off, I got to the airport. I relaxed in a lounge over beers with Kevin Fenzi, Jan Zeleny, and Stephen Tweedie, before we went to our respective flights. Then after a quick flight home, it was the usual “fun” making my way down I-95 from the airport to home. Monday morning was right around the corner…
Here’s to another great Flock, and to doing it again next year!
Fedora’s quality makes complacency easy. But in truth, we’re always under construction — or we should be. You could call that constant disruption by different names. Risk positive. Forward leaning. Embracing change. Since inception, Fedora was intended to avoid the status quo. So what’s next for shaking up said status?
As you’re probably aware, starting with Fedora 21 our releases are a bit different. We now have different editions, each serving a specific type of consumer. The editions today are Workstation, Cloud, and Server. The editions differ in mostly minor ways, though, and we build them mainly the same way. That’s why these editions aren’t the end goal of change. Rather, they’re a step in changing what Fedora releases.
Matthew Miller, Stephen Gallagher, and others have been steadily laying out a vision for change in Fedora. Admittedly, that vision’s high level. It uses words like “Rings” to simplify and amplify concepts that are about change. These concepts are well attuned to higher level Fedora goals. While editions can effectively broaden Fedora’s appeal to different consumer audiences, we also want to attract more makers of things.
These days, the largest maker audience is beyond just those who care about building a platform. As Red Hat’s Paul Cormier said last year, “The application is king.” Makers of applications and things above the OS are always on our minds. So how does this relate to the Fedora release of the future?
For many years now, we’ve been building our release essentially the same way. We take a big bag of packages, like building blocks, and glue them together in a group that makes sense. We wrap that group in a shippable format (or several), sometimes with an installer, validate, and release it. Lately many Fedora folks are thinking about what that future release looks like, though. I would offer that the future release should be some combination of a strongly managed center, curated stacks, and an expanding nebula of containers.
Managed center? Does that mean the return of Fedora Core? No. Forget about Fedora Core. It’s dead, and it’s not coming back. Having the central part of the OS carefully managed in the community isn’t the same thing. Fortunately, there are emergent tools to do this. Among them, Project Atomic looks like one of the best bets in the space we care about. Atomic makes shipping an integrated, validated set of content easier. That content still comes from the packages we know well — kernel, glibc, bash, and others. But the rpm-ostree basis of Atomic can prevent slew in an installed system.
What do I mean by “slew”? Right now, the only Fedora release we know well is the one we put out at GA. After that, all bets are off. Any user potentially has a random subsets of updates on top. Saying “I’m using Fedora 22” is not very meaningful soon after release. That slew also makes validation, troubleshooting, diagnosis, and recovery unnecessarily hard. What if we manage this central content more carefully, using a model like rpm-ostree? We could validate a release more regularly, at least at a basic functional level.
There are other benefits as well. What if we constructed Rawhide using the rpm-ostree model? Imagine that an important core piece of the OS broke. We’d want to file that bug, track, fix, and update. Right now, hundreds of developers have to manually, sometimes painstakingly, fix their systems. This wastes large amounts of aggregate time. This problem is largely why many interested developers don’t run Rawhide as much as would be helpful. In the new model, if you’re invested in the problem, you can still work on it, of course. But the rest of the Rawhide users could, with a single command, back out to the previous tree and keep working. This keeps more interested (and interesting) people using Rawhide.
You could probably make a case for file system snapshots here. I think those are still useful for user data in this model. It’s not clear that snapshots would solve the slew problem above without imposing restrictions on them in some fashion. Would users be happy with that? Hard to say.
So what about these curated stacks? Well, to be honest it’s not yet clear what tech wins here. On one hand, enhancing rpm-ostree to allow layering might be a way forward. Currently rpm-ostree is somewhat monolithic in that you can’t really mix or match stacks readily… yet. My understanding is the Atomic guys are thinking about this problem already, though, and how to solve it. So I expect the code will catch up to (and maybe overtake) the concept before we know it.
Alex Larsson is also doing interesting work on what he calls “sandboxed apps.” Sandboxing in this model might not be too different from containers. The concepts seem quite similar. Throw into this mix the recent progress on overlayfs, which is now part of the mainline Linux kernel. What you have is ripe ground for a new method to build and release big swaths of a platform.
Again, we have building blocks of a solution for interesting problems, some long-standing. The problems of the central core above are shared at common application layers as well. But it’s useful to detach them at key dividing lines for obvious reasons. In some way, shape, or form, it seems inevitable that Fedora must take a swing at these problems, even if it’s on a per-edition basis.
This leaves the nebula of containers I mentioned. A year and a half ago, containers were all the rage. People were thinking about how they affected the technology landscape for infrastructure. Perhaps some people reading this were thinking about containers as a fad. Time and consumers and commercial customers have pretty much proved that wrong. Containers allow app developers and users to move swiftly (or not) independent of the OS. The technology is here to stay, because it mostly makes OS maintenance issues someone else’s problem. In this case, ours — Fedora’s.
For a long time we’ve relied on people working the application layer to radically change their methods if they’re interested in deploying on Fedora. But frankly, time has shown the world doesn’t care to do that. So our choice is to adapt or face irrelevance. Matthew Miller has spoken to this point several times, so I won’t belabor it. My only other point is that, however we build the future Fedora release, it should make King App comfortable.
For those people who might ask why we should take on this work, I’d start with a couple of thoughts. First, what’s in it for me as a contributor? Well, it depends first on whether you care about an edition that might use this model. I’m pretty involved in Workstation edition. For some time we’ve been interested in how to update Fedora for consumers at the OS and application level, rather than packages. And something like a core + containers model using Atomic directly solves that problem. The Cloud SIG already has an Atomic image designed primarily for container management. Their user base has different expectations from Workstation. But clearly there’s room for great collaboration here, and I expect for Server too.
Another reason this is interesting is app vendor involvement in Fedora. Containers abstract the distribution problems away from application vendors. We know Fedora is in a lot of embedded hardware and other projects in the Real World. That problem space fits well with our current platform construction model. As I said before, we don’t necessarily need to stop doing that. But at the same time, embracing the rise of the app container lets us more effectively reach the developer audience. This is not just about our Workstation edition but, more generally, people who build and make interesting things. This is also somewhat tied up with the implementation of Rings. That’s why I look forward to Matthew and Stephen driving more detail and progress on that front.
Finally, by solving this problem we can effectively influence the value of future RHEL, as well as CentOS and our other family members. That value is one reason Red Hat invests time and resources in our community. Making changes to grow that value is always a win-win for Fedora as well.
The hero’s journey
So, does all this mean we won’t have live or installation ISOs in the future? Not necessarily. They could still be useful. It would depend, as most things do, on what it takes to validate and maintain all those things in the long run. For example, I don’t see this idea necessarily affecting spins. Communities around those releases are accustomed to how they build their bits. But I think at least one (maybe more) of the official Fedora release editions will need to opt for this new model if we’re to make progress.
The entire set of changes needed isn’t yet clear, of course. One thing that is clear: release engineering process is, by definition, central. And there’s no doubt we’re looking at a healthy chunk of work. But I believe strongly enough in the possibilities that our team has an extra full-time person now to drive planning of those changes, consult with the Fedora release engineering team, and build a community around the tools needed. We are going to emphasize modularity, so the winning technology of tomorrow can plug in to releases down the road. The initial goal I’d like to set is to prototype a release of at least one Fedora edition for F23, and be part of F24 official release at GA.
That means it’s going to be an exciting year of construction ahead for Fedora. So please excuse our dust!
This release has been a long time coming. It has been about a year since F20 release, and the pause we took as a community to embark upon the first steps of Fedora.next. I know many people have been anxious for the pause to be over. Finally the day has come and gone, and the release seems to be hitting on all cylinders!
I wanted to say thanks to the whole community that contributed to Fedora 21 release. It’s impossible to name everyone who helped, and if I leave someone out it might disappoint someone. So let me just say to everyone:
David Gay pointed me to an interesting project called Git by a Bus. Git by a Bus analyzes your git repository and attempts to quantify risk of having lots of code knowledge tied up in only a few people. Git by a Bus does its analysis by going through the repo history and making an estimate of what it calls unique knowledge.
This project blog page describes the analysis and metrics used. Perhaps this is a useful way to show how Fedora is doing as a project, across repositories like our web applications and infrastructure. It might show where we need to encourage further community development and participation so we avoid the “eaten by raptors” problem.
You might recall that “eaten by raptors” is Fedora shorthand for “hit by a bus” (violent idiom) or “going to work for another company” (not always applicable to Fedora, although certainly to Red Hat as a major contributor). We try to solve this problem by spreading project knowledge and documenting our processes. That way, if someone was eaten by velociraptors, the project can keep going without too much of a disturbance. This problem is common to any team or enterprise, not just open source. But I like to think our velociraptor spin is unique.
Here’s an example output I prepared for the MirrorManager project, which we use to provide content to Fedora mirrors worldwide. This is a potential example of high risk. One developer (the inimitable and awesome Matt Domsch) has unique knowledge of this project that is at risk if velociraptors manage to track and eat him. No doubt Matt would put up a good fight, but as you probably know they are clever girls.
Thankfully, there is a MirrorManager related Fedora Activity Day happening later this year. During that time the Fedora infrastructure, release engineering, and applications teams hope to accumulate and document more MM-related knowledge. At the same time they’ll be using this knowledge to architect, plan, and further develop the next revision of MirrorManager.
If you’re a principal in an FOSS project using git for your code, you might find Git by a Bus useful.
We’ve had a long release cycle for Fedora 20 to accommodate a lot of thought and planning. How do we get three products out in place of one? How will we build them? What needs to change? How do we get the bits into place for releases? It’s a lot of work, and we’re not done yet. I suspect that we’ll see further change in the Fedora 22 cycle — although I’d also bet we won’t want to extend another cycle for it.
For my part as manager of the Fedora Engineering team, I am proud of the work all the folks on the team have done to support Fedora 21 Alpha. From changes to infrastructure, to work on new web applications to support multiple products, to notifying Fedora Project members of activity and contribution, to making things generally more beautiful, the team is tireless in their effort to serve the community. As always, my hat is off to them with awe and inspiration.
And of course it’s also off to you, the many, many members of the Fedora Project overall. From Ambassadors to Marketing to Docs to Translation to Websites to… whew. I ran out of breath there. But all of you folks rock!
If you want to pick a copy of any of the new Fedora products — Fedora Server, Fedora Cloud, or Fedora Workstation — just visit the prerelease download page featuring Fedora 21 Alpha, and take your pick.
One of the aspects of Fedora is holding public meetings on IRC. We use Meetbot (courtesy of Debian, thanks!) to help administer meetings. Common commands allow Meetbot to do all the hard work of recording proceedings. The automatic minutes make it possible for people who couldn’t attend to follow what happened in the meeting. These minutes are key for maintaining transparency and information flow around the project.
But the minutes still depend on the people who chair the meetings to use the command set to record important data.
#startmeeting – Sets the overall group for the meeting
#endmeeting – Cleans up when done, and gives you the URLs for the minutes
#topic <Topic name> – Sets a topic heading for the next portion of the meeting
#info – Record some information that’s useful for anyone reading the minutes
#action <nick> <thing to do> – Clarifies who’s got the ball to complete something before the next meeting; it’s usually a good idea to set a due date*
#agreed – Documents something the attendees agreed on, also important to make decisions transparent
#idea – Helps give visibility to something no one is doing yet, but could be useful (also see #help in the MeetBot page)
#chair <person>… – Add someone(s) to the list of people MeetBot will listen to for commands
* A good friend of mine pointed out that unless you set a due date for an action item, you’re not writing actions, you’re writing a wish list. It should not only be clear who’s got the ball, but when they’re expected to give it back.
Here’s an example of a meeting I ran recently where I used the MeetBot commands to record useful minutes. If you were to look at these minutes later you’d get a pretty good idea of what was covered. You’d also know who was supposed to do tasks before the next meeting. There are a couple action items without clear dates, which is sub-optimal. But overall the meeting minutes are pretty clear.
In some cases, I ended up repeating things people said, using the #info command at the front to tell MeetBot to record in the minutes. If you’re running a meeting you should be prepared to do this. I also like to add everyone in the meeting to the #chair list, to help increase information flow when needed. (It’s also not a bad idea to reduce the chance that a single chairperson will be knocked offline and unable to #endmeeting.)
Are you reading your minutes when done to see if they’re effective? If not, you should. Use what you find to make your meetings better and more transparent for the community. I thought about showing some recent examples of poor minutes usage, but I didn’t want to embarrass anyone.
If your minutes only serve to show a link or two, and an attendance roster, that’s pretty much useless for most community members. Sure, logs are useful, and good for transparency too. But it takes a long time to read logs and extract necessary points from the dialogue. That dialogue can also sometimes be confusing after the fact due to the way IRC works.
Use the facilities we have available to us in Fedora to provide more information and transparency on what you’re doing. The couple of extra minutes per meeting spent using MeetBot will save each reader many more in return!
Here’s a summary of Saturday’s activity at Flock 2014 where I participated or attended. I also have blog entries for Day 1, Day 2, and Day 3.
The constant stream of late nights was really getting to me. Didn’t arrive at the venue until about 9:15am. I skipped the first session and had some coffee, courtesy of Smooge.
Caught up on email sent overnight from people in the USA, and did final preparation for my talk.
I gave my session on the connection between RHEL and Fedora. I also discussed how well things went for RHEL 7 due to work in the Fedora community. I feel like it went very well. You can watch the complete video here.
I had an excellent conversation with Alberto Ruiz, who manages Red Hat’s desktop applications team.
Sat in the hall with Patrick and got a Taskwarrior server running on one of my boxes.
Joined the session on revamping governance in Fedora, which was run by Toshio Kuratomi and Haïkel Guémar. This was hands down the best accomplishment of Flock. There will be a proposal for Board revamp coming from this session (finally!). I’m looking forward to the ensuing discussion and resulting improvements.
At this point I was finally exhausted. I headed back to the hotel early to do a little more reading and writing. I met up with some of the Anaconda team for a late dinner. Then I packed so I’d be ready in the morning to catch my flights back to the USA.
The Flock conference was excellent this year. It was nice getting back into the swing of community things. I enjoyed meeting up with everyone I saw. If I didn’t get a chance to see and talk with you personally, I’m still glad you were there. I hope you had a great time at Flock in Prague. Let’s do it again next year in the USA!
Got lunch late, ending up at a table with Stephen Tweedie and a few others. We talked about containers and strategy.
Touched up my slides for Saturday, getting straight in my head how I wanted the presentation to go. Reveal.js is cool.
Attended Richard Hughes’ session on building an application installer. GNOME Software is a huge step in usability, and it was enlightening seeing the huge amount of work that went into this tool. I wrote an article on Fedora Magazine covering this presentation.
Attended Ralph Bean’s excellent workshop on making tools with fedmsg, the Fedora messaging bus built on Zeromq. We learned how to use just a few simple lines of Python to build a Twitter feed from Fedora Badges. Amazing!
Attended the workshop on DevAssistant. I talked with the developers to learn about their future plans and to discuss desktop integration.
Have you seen Máirín Duffy’s post on the Fedora Design team’s next-generation design for the Fedora Project website? It’s a brilliant design based around the idea of a “sub hub.” These screens help customize the website to fit different sub-communities, initiatives, teams, or projects.
Máirín published this post with the design mockups a few weeks ago and they’re still open for feedback. I love the concept and the mockups, and the way it brings the site a little closer to the functionality people expect for interaction in other communities. The sub hub design offers a sites not just for promotion, but also for bringing people together for communication and information.
I’m sure the Design team will move forward at some point to bring these concepts into reality. But before they do, I know they’d appreciate hearing from community members. Even just offering feedback that “This is awesome!” is useful, so the designers know there is a solid mass of people who like the work. You can visit the site here to offer your constructive comments.
If you don’t, that’s OK too! Just be polite and specific in your comments. Rather than saying how to fix something, talk about why something doesn’t work for you. Designers are good at figuring out how to solve usability problems once they know more about the effects on the user. Those of us without a lot of design and usability experience often suggest solutions that seem like they’d work, but really might cause more problems for other users. So it’s best to concentrate on symptoms.
So if you haven’t checked out the post and offered some constructive feedback, please feel free. I’m really looking forward to seeing how things move forward with these designs and hope you’re excited about them too.
If you’re excited enough about the work to get involved and help, the Design team would love your contributions. There’s also more information about how to contribute here. There are several repositories set up where you can test existing ideas, change them with your own, and contribute changes back to the team using a pull request. Getting involved is easy, and the Design team is famous for their friendliness and willingness to help people get started contributing. So don’t be afraid to jump in!