Last week I attended the Red Hat Summit 2018. There I interacted with customers, partners, community leaders, and some friends from around Red Hat. I enjoy going every year and look forward to it, despite the exhaustion factor. This year included a fun event for Fedora — a birds of a feather (or BoF) session. Read on for my report.
Along with several people on the Fedora Engineering team, I recently attended the DevConf.cz 2017 event. The conference has grown into an amazingly successful gathering of open source developers. Most attendees live in Europe but there were some from every continent. The coverage spanned all the big open source buzz-generating technologies. Session topics included containers, PaaS, orchestration and automation, and DevOps.
One of the most frequent topics I heard at and around the conference was CI — continuous integration. The idea of CI isn’t new in and of itself. Many open source projects put it to use already. If you do any work on Github, you’ve already seen it at work. (Also notably, OpenStack features one of the biggest and arguably most successful at-scale efforts.) However, it hasn’t taken strong root in Fedora, yet.
Based on the discussions I had and heard at DevConf.cz, that’s likely to change soon. Why? Well, for one thing, Fedora still breaks too often. This is especially (but not exclusively) true about Rawhide. But in many cases, we can easily understand these breakages. That means we can detect and prevent them before they occur. While update testing in Bodhi helps in the case of stable releases, that’s not a cure-all. That process still depends heavily on manual efforts that aren’t guaranteed. Moreover, those efforts form a gate of several days at a minimum, between building, karma, and tagging. This process no longer scales with new, fast-moving deliverables like Fedora’s edition of the Atomic Host. One size no longer fits all. We need a more flexible, automated approach.
What does that mean? For Fedora, the CI concept means gating on automated tests that guarantee some level of validity to a change. These tests are run prior to introducing changes into a tree, ostree, module, container, etc. That way, we don’t pass breakage on to the consumer. But we can message that breakage back to the maintainer directly. The maintainer then quickly acts to fix the issues. And they see more immediate results from their fix.
We must do some work in the Fedora infrastructure to make this process work for Fedora contributors (maintainers especially). Some app work is also required to support our packaging and deliverables. We must make sure it’s easy for people to contribute to tests. Maintainers need to easily understand the status of their build requests. And of course the community must be able to continue contributing to higher levels of testing that make for a better Fedora. So CI isn’t the end of the story. It’s just another strong link in a chain of improvements.
Fortunately, we’re well situated to do great work here. For instance, the Fedora Infrastructure team already plans to front our dist-git package control with Pagure. (You probably know Pagure is a full-featured collaborative git forge.) This ensures the Fedora dist-git acts as a source for CI, tested with each new proposed change. But we believe this is unlikely to cause more complexity. Packagers can expect to use tools they already know. Furthermore, they could use the popular, effective pull request model to maintain packages and develop testing. This lowers the bar to contribution while maintaining high quality.
Another point in our favor is the entire Fedora application infrastructure has been running on the fedmsg messaging bus for years. We already can orchestrate and coordinate a huge number of automated activities around events involving our apps. Therefore a higher level of continuous integration and testing is well within our grasp.
Of course, this blog post is light on details. (What else would you expect from a manager?) 😛 Those details are obviously important. As the least technical person in the Fedora Engineering team, though, I’m the last person you’d want to describe them in detail. But the folks in the Infrastructure community team will be launching a series of discussions on the mailing list to explore those details.
We’re all passionate about making Fedora better. So hearing feedback from our contributors and colleagues at DevConf.cz has us very excited and enthusiastic about this next level. I encourage you to get involved and contribute constructively to the discussions!
I realized that some folks around the Fedora community may wonder why they don’t see me around as often this week and next week. I’m still alive and well, but I’m traveling in the Czech Republic. I’m currently in Brno for some Red Hat departmental meetings. Following that, I’ll be attending the Devconf.cz event. Then I’ll be back in the Brno office for a few days of other work and team meetings. I’ll be back in my home office on Monday, February 15th and around as usual at that point.
Getting started at Flock
Like everyone on the Fedora Engineering team, I was in Rochester for the Flock conference last week. After several flight delays on our direct flight from DCA to Rochester, Justin Forbes, Ricky Elrod, and I finally arrived a little after 9:00pm — about four hours late. Thankfully Josh Boyer came to pick us up at the airport.
Flock had a team of organizers within OSAS (and Josh also assisted throughout). As a former FUDCon organizer, though, I know the value of extra hands showing up to do work. Since old habits die hard, I showed up expecting to help out behind the scenes. That means I didn’t get to see a huge amount of content I was personally interested in. But in return hopefully everyone had a smoother Flock experience, especially speakers.
When I arrived, I reported to Tom Callaway, Ruth Suehle, and Josh. They got the conference rooms opened, and I helped set up the speaker workstations. We worked pretty late, well after midnight. Things were looking a little bleak at that point, with execrable network bandwidth, no projectors, no screens, and no audio for the ballroom.
Fears and worries abate
Nevertheless, the next morning Josh and I got up early and grabbed coffee at nearby Tedward’s. This place was a godsend, although their 7:00am opening time forced us to walk around a bit until we could get in.
We went down to do some additional setup. The organizers had worked with Remy DeCausemaker to get a bunch of loaner projectors from RIT so we’d be ready for the first sessions at 10:00am. (EDIT: According to Remy, Tim Duffy and Dan Schneiderman are the heroes of this particular day; see comments below.) So at least our speakers would be in OK shape. I helped Josh and Tom get everything ready in those rooms, while Ruth made sure registration and other logistics were under control. I missed Matthew Miller’s keynote, but I’d seen at least some of the material previously.
After lunchtime, things continued to drastically improve. The rental projectors showed up, along with small screens for each room and big speakers for the ballroom. The wireless internet improved quite a bit when a switch flip occurred due to our conference starting up. (It was dismal Tuesday night!) We had all the speakers trained on how to record their talks locally, to get around the constrained network bandwidth.
Suddenly things were looking up! Not surprisingly, the Fedora Engineering team dinner that night at The Old Toad was much more enjoyable. Since I wasn’t overly worried about the conference experience for the speakers and attendees any longer, it was easier to relax and enjoy the company of the team. I was so happy that we were able to get together in one place, since we really only get to do that once a year. (Incidentally, our friend Stephen Smoogen was absent from Flock due to family commitments — we missed you, Smooge!)
I continued to monitor speaker rooms most of Wednesday and Thursday. I managed to make it to a couple sessions where I wasn’t sure there would be any senior Fedora leadership around. For example, I attended the Fedora Magazine session by Chris Roberts as well as most of the Fedora Hubs session by Máirín Duffy and Meghan Richardson.
I attended and loved Major Hayden‘s (of Rackspace fame) Thursday keynote on fighting impostor syndrome. It was one of the most practical that I’ve seen on this topic. I feel impostor syndrome is just a fancy way to refer to insecurity, a common trait for conscientious people. But that doesn’t make the strategies Major outlined any less useful or thoughtful. He gave a great talk — engaging and humorous without diluting the material. If you have a chance to invite him to a conference to speak, definitely do so!
I gave my own talk on Remote Ninjutsu on Thursday afternoon. The slides for the talk are here, although the video will be more useful for context. All the Flock 2015 videos are supposed to be available at some point in the next couple of weeks. Stay tuned for announcements about them.
The Thursday night social event at the Strong National Museum of Play was fantastic. It was a great way to blow off steam and enjoy the company of fellow Fedorans. I’m not sure how the organizers managed to find such a perfect venue!
Workshops and Flock wrap-up
On Friday I enjoyed the keynote by Jon Schull of eNable, the community that is flipping the script on prosthetics provision through 3D printing. It was a very moving look at how people are applying open source to make the world better for people in need.
Then the workshops beckoned. Now that I’d finished my Flock duties helping speakers and attendees, I was able to attend several sessions that were relevant to me personally, including the Fedora/CentOS rel-eng joint session, and my own on revamping the Flock software stack.
Once again, the Friday night social event at the George Eastman House was marvelous. It was a beautiful, grand mansion and the tour was quite interesting. I’d love to go back there sometime to see the exhibits I missed!
Flock conferences are always especially great for their hallway track. So many discussions can be had or progressed that way with high bandwidth. The challenge is always to move that discussion to a transparent context if it involves people not present, though. I’ve been seeing many trip reports from people’s blogs about Flock, and resulting list discussions, so I think that process is well underway.
Of course, that means Flock is a very engaging event. It takes a lot of attention and brainpower to shift focus for all those conversations! As a result, by Saturday afternoon I know I was fairly exhausted — in a good way, though. Several other people I know felt likewise, and commented on how well the conference had gone. In fact, I heard a number of comments that this was the best Flock, and even Fedora premier event, yet. The OSAS folks deserve special recognition for pulling off a fantastic conference.
Sunday started with a couple meetings, including with Matthew Miller and Jan Ku?ík, our new Fedora program manager. Then, after seeing a few other friends and colleagues off, I got to the airport. I relaxed in a lounge over beers with Kevin Fenzi, Jan Zeleny, and Stephen Tweedie, before we went to our respective flights. Then after a quick flight home, it was the usual “fun” making my way down I-95 from the airport to home. Monday morning was right around the corner…
Here’s to another great Flock, and to doing it again next year!
As you might have seen, the Red Hat Summit is happening this week. As usual, I’m here volunteering on the staff, helping with training, the booth, manual labor, and whatever else I can do to assist our awesome event gurus. I love seeing and meeting customers and partners at this event, so I look forward to it every year.
Of course, it comes at a price. I’m not really around on IRC and slow to get to email compared to a normal work week. To make matters worse (or better, depending on your perspective), I’ll be on vacation for about a week and a half following the event. For that reason I put my outage on the vacation calendar on Fedocal.
Now, I don’t think this will unduly impact anything in Fedora. There’s little that depends on me personally since mainly I just try to help things happen for my teammates. But I wanted to make sure the community is aware and knows what to expect.
Are you (1) coming to Flock 2015; (2) located, or planning to be, in the Boston, MA or Westford, MA area; and (3) interested in taking a free, power and wifi-equipped bus if offered?
If so, then pre-register by 1900 UTC Friday, June 5, and make sure you indicate in your registration you are located in the Boston/Westford area. I’ll collect the final number of these signups on Friday afternoon US/Eastern time. We’ll use that number to determine whether or not it’s economical to book the bus.
The announcement about the bus will be made next week, so there will be plenty of time to make appropriate travel plans either way.
I’m here in Westford over the weekend with other members of the Fedora Design Team for their Fedora activity day. I traveled up to chilly Boston on Thursday midday, so I could assist by transporting people from the airport to our hotel in Westford.
On Friday, we convened in the Red Hat office in Westford. As is usual for my Westford visits, I pretty much spent the day running from place to place taking care of things unrelated to the Design team, but which needed to be done in my role as a manager. Fortunately most of the day was spent by the team figuring out policy and processes. I don’t feel like my input was needed there, or even appropriate since I’m not a frequent contributor to design tasks — as much as I love the team! So it all worked out for the best, I think.
Today, though, I was able to contribute. One of the major task areas was to do issue triage and fix up the team’s Trac instance. That was something I was (somewhat) qualified to do. I helped the team go through all the pending tickets, closing stale tickets (no response, unclear goals, redundant, etc.).
Then, using the categories developed by the team on Friday, I helped update the parameters on the Trac instance to match. I also set up a bunch of reports on Trac to match relevant agenda items for the reboot of the Design team meetings. This way, they can call up a set of reports to follow up on tickets methodically.
I even learned some good SQL-fu thanks to my buddy Langdon White. I finally got a start shedding my misunderstanding of joins, so I can do more complicated queries. One of the results was this report, which tells the team when an issue reporter is not responsive to questions, in line with the processes the team worked up on Friday.
People tried hard to make sure the FAD was remote accessible, so if you couldn’t be here, you could still monitor or participate. It was difficult to keep some of the facilities working. For instance, this afternoon I discovered that someone had bounced us from our own room on OpenTokRTC. That made it, well, rather difficult for us to broadcast there. I hope remote attendees will understand the difficulties and be confident we tried to make this a decent remote event.
Tonight I’ll probably think about some additional reports, and then do a little personal work. Tomorrow will be some more work sprints until I start chauffeuring people to the airport after lunch. I was happy to be part of the group and to help the participants have an effective FAD.
- The constant stream of late nights was really getting to me. Didn’t arrive at the venue until about 9:15am. I skipped the first session and had some coffee, courtesy of Smooge.
- Caught up on email sent overnight from people in the USA, and did final preparation for my talk.
- I gave my session on the connection between RHEL and Fedora. I also discussed how well things went for RHEL 7 due to work in the Fedora community. I feel like it went very well. You can watch the complete video here.
- I had an excellent conversation with Alberto Ruiz, who manages Red Hat’s desktop applications team.
- Went with Alberto and Patrick Uiterwijk to lunch at the cafeteria. Got to know Patrick a little better, since he will soon be joining us on the Fedora Engineering team.
- Sat in the hall with Patrick and got a Taskwarrior server running on one of my boxes.
- Joined the session on revamping governance in Fedora, which was run by Toshio Kuratomi and Haïkel Guémar. This was hands down the best accomplishment of Flock. There will be a proposal for Board revamp coming from this session (finally!). I’m looking forward to the ensuing discussion and resulting improvements.
At this point I was finally exhausted. I headed back to the hotel early to do a little more reading and writing. I met up with some of the Anaconda team for a late dinner. Then I packed so I’d be ready in the morning to catch my flights back to the USA.
The Flock conference was excellent this year. It was nice getting back into the swing of community things. I enjoyed meeting up with everyone I saw. If I didn’t get a chance to see and talk with you personally, I’m still glad you were there. I hope you had a great time at Flock in Prague. Let’s do it again next year in the USA!
- Didn’t make it up quite so early today, due to not turning in until about 2:30am the previous night. I got to the school basically on time, but worked on email and day job stuff for a little while.
- Attended Matthew Miller’s joint session on Fedora.next.
- Got lunch late, ending up at a table with Stephen Tweedie and a few others. We talked about containers and strategy.
- Touched up my slides for Saturday, getting straight in my head how I wanted the presentation to go. Reveal.js is cool.
- Attended Richard Hughes’ session on building an application installer. GNOME Software is a huge step in usability, and it was enlightening seeing the huge amount of work that went into this tool. I wrote an article on Fedora Magazine covering this presentation.
- Attended Ralph Bean’s excellent workshop on making tools with fedmsg, the Fedora messaging bus built on Zeromq. We learned how to use just a few simple lines of Python to build a Twitter feed from Fedora Badges. Amazing!
- Attended the workshop on DevAssistant. I talked with the developers to learn about their future plans and to discuss desktop integration.
- Met up with Garrett LeSage, Chris Roberts, Matthew Miller, Haïkel Guémar, and others for a great dinner at an Italian pizzeria. It was delicious.
- Late hangout with friends kept me up yet again too late!
Here’s a summary of today’s activity where I participated or attended:
- Up at 5:45am so Matthew and I could meet up with Josh Boyer, Tom Callaway, Ruth Suehle, and Joe Brockmeier for breakfast. Then we arrived at the Flock venue early.
- Helped set up rooms with wifi information for attendees. Discovered the rooms feature electronically controlled windows. Once opened, these made the venue much more comfortable.
- Missed keynotes myself while ushing people around to them. 🙂
- Worked on my slides for Saturday’s talk, in the great Fedora tradition of iterating until the last minute.
- Sat in on Tim Flink‘s Taskotron talk, and took notes for a Fedora Magazine article.
- Went to a lunch meeting with Ludek Smid, Jaroslav Reznik, Joe Brockmeier, and Matthew Miller. We discussed some project management assistance for our Atomic/OStree work in Fedora. Very productive and we also had a good time.
- Sat in on Christian Schaller’s Fedora Workstation talk. It was very well attended, so I think the idea that the Linux desktop is dead might be a tad premature. 😉
- Sat in on Marina Zhurakhinskaya’s talk on the Outreach Program for Women. I’m happy to say Fedora is an active player in this space. I look forward to our doing even more.
- Sat in on a talk on Waartaa by Ratnadeep Debnath and Sayan Chowdhury. This is an interesting take on a Web IRC client as a basis for other collaboration tools.
- Sat in on Chris Roberts’ and Marie ‘riecatnor’ Nordin’s talk on Fedora Badges and badge design. (If you’re looking for the resources shown in the talk, look here.)
- Headed back to the hotel to finish a Fedora Magazine article. Then I met up with friends to head over to our event at The Pub.