Tag Archives: infrastructure

Flock 2015 thoughts.

Getting started at Flock

Like everyone on the Fedora Engineering team, I was in Rochester for the Flock conference last week. After several flight delays on our direct flight from DCA to Rochester, Justin Forbes, Ricky Elrod, and I finally arrived a little after 9:00pm — about four hours late. Thankfully Josh Boyer came to pick us up at the airport.

Flock had a team of organizers within OSAS (and Josh also assisted throughout). As a former FUDCon organizer, though, I know the value of extra hands showing up to do work. Since old habits die hard, I showed up expecting to help out behind the scenes. That means I didn’t get to see a huge amount of content I was personally interested in. But in return hopefully everyone had a smoother Flock experience, especially speakers.

When I arrived, I reported to Tom Callaway, Ruth Suehle, and Josh. They got the conference rooms opened, and I helped set up the speaker workstations. We worked pretty late, well after midnight. Things were looking a little bleak at that point, with execrable network bandwidth, no projectors, no screens, and no audio for the ballroom.

Fears and worries abate

Nevertheless, the next morning Josh and I got up early and grabbed coffee at nearby Tedward’s. This place was a godsend, although their 7:00am opening time forced us to walk around a bit until we could get in.

We went down to do some additional setup. The organizers had worked with Remy DeCausemaker to get a bunch of loaner projectors from RIT so we’d be ready for the first sessions at 10:00am. (EDIT: According to Remy, Tim Duffy and Dan Schneiderman are the heroes of this particular day; see comments below.) So at least our speakers would be in OK shape. I helped Josh and Tom get everything ready in those rooms, while Ruth made sure registration and other logistics were under control. I missed Matthew Miller’s keynote, but I’d seen at least some of the material previously.

After lunchtime, things continued to drastically improve. The rental projectors showed up, along with small screens for each room and big speakers for the ballroom. The wireless internet improved quite a bit when a switch flip occurred due to our conference starting up. (It was dismal Tuesday night!) We had all the speakers trained on how to record their talks locally, to get around the constrained network bandwidth.

Suddenly things were looking up! Not surprisingly, the Fedora Engineering team dinner that night at The Old Toad was much more enjoyable. Since I wasn’t overly worried about the conference experience for the speakers and attendees any longer, it was easier to relax and enjoy the company of the team. I was so happy that we were able to get together in one place, since we really only get to do that once a year. (Incidentally, our friend Stephen Smoogen was absent from Flock due to family commitments — we missed you, Smooge!)

Fedora contributors at Flock gather at Victoire for dinner
Fedora contributors at Flock gather at Victoire for dinner

I continued to monitor speaker rooms most of Wednesday and Thursday. I managed to make it to a couple sessions where I wasn’t sure there would be any senior Fedora leadership around. For example, I attended the Fedora Magazine session by Chris Roberts as well as most of the Fedora Hubs session by Máirín Duffy and Meghan Richardson.

I attended and loved Major Hayden‘s (of Rackspace fame) Thursday keynote on fighting impostor syndrome. It was one of the most practical that I’ve seen on this topic. I feel impostor syndrome is just a fancy way to refer to insecurity, a common trait for conscientious people. But that doesn’t make the strategies Major outlined any less useful or thoughtful. He gave a great talk — engaging and humorous without diluting the material. If you have a chance to invite him to a conference to speak, definitely do so!

I gave my own talk on Remote Ninjutsu on Thursday afternoon. The slides for the talk are here, although the video will be more useful for context. All the Flock 2015 videos are supposed to be available at some point in the next couple of weeks. Stay tuned for announcements about them.

The Thursday night social event at the Strong National Museum of Play was fantastic. It was a great way to blow off steam and enjoy the company of fellow Fedorans. I’m not sure how the organizers managed to find such a perfect venue!

Workshops and Flock wrap-up

On Friday I enjoyed the keynote by Jon Schull of eNable, the community that is flipping the script on prosthetics provision through 3D printing. It was a very moving look at how people are applying open source to make the world better for people in need.

Then the workshops beckoned. Now that I’d finished my Flock duties helping speakers and attendees, I was able to attend several sessions that were relevant to me personally, including the Fedora/CentOS rel-eng joint session, and my own on revamping the Flock software stack.

Once again, the Friday night social event at the George Eastman House was marvelous. It was a beautiful, grand mansion and the tour was quite interesting. I’d love to go back there sometime to see the exhibits I missed!

The music parlor in the George Eastman House
The music parlor in the George Eastman House

Flock conferences are always especially great for their hallway track. So many discussions can be had or progressed that way with high bandwidth. The challenge is always to move that discussion to a transparent context if it involves people not present, though. I’ve been seeing many trip reports from people’s blogs about Flock, and resulting list discussions, so I think that process is well underway.

Of course, that means Flock is a very engaging event. It takes a lot of attention and brainpower to shift focus for all those conversations! As a result, by Saturday afternoon I know I was fairly exhausted — in a good way, though. Several other people I know felt likewise, and commented on how well the conference had gone. In fact, I heard a number of comments that this was the best Flock, and even Fedora premier event, yet. The OSAS folks deserve special recognition for pulling off a fantastic conference.

Sunday started with a couple meetings, including with Matthew Miller and Jan Ku?ík, our new Fedora program manager. Then, after seeing a few other friends and colleagues off, I got to the airport. I relaxed in a lounge over beers with Kevin Fenzi, Jan Zeleny, and Stephen Tweedie, before we went to our respective flights. Then after a quick flight home, it was the usual “fun” making my way down I-95 from the airport to home. Monday morning was right around the corner…

Here’s to another great Flock, and to doing it again next year!

Flock attendees wind down after the conference ends... with more hacking!
Flock attendees wind down after the conference ends… with more hacking!

DevConf.cz, days 1 and 2.

DevConf.cz day 1, Friday.

Friday was the first day of sessions at DevConf.cz, the biggest and best Czech open source event by developers, for developers. The event was packed, with over 900 attendees even before the weekend started!

First up at 9:00 sharp was Tim Burke’s keynote about how Red Hat sees the IT market, specifically Linux and open source technologies. He covered how the various pieces of cloud, applications, storage, and platform fit together. It was pretty breakneck because there wasn’t a lot of time until the sessions started, but well observed and thoughtful. It’s clear the technologies built by people at this conference will set the pace for the future. The market has placed its bets on Linux and open source, and now it’s on us to deliver!

Langdon White followed with a story of startups. He covered how the tradeoffs between agility, stability, and maintenance can be mitigated by Software Collections. Software Collections allow IT groups to add stacks on their platform without affecting the deployment itself, while meeting more needs for developers and users.

Alex Larsson did a talk to a packed room (the biggest at the conference, no less!) on Docker, the open source container engine rapidly sweeping the community with its speed and flexibility. Fedora is rapidly developing a great grasp of Docker, and you can already install it on all supported Fedora releases. Obviously Red Hat has taken a huge interest in Docker too, so it’s no surprise the talk was SRO.

I went to Colin Walters’ session on OStree, a new way of distributing Linux operating systems. I found this session incredibly compelling, and I hope we look seriously at OStree in Fedora because of the problems it solves. There are clearly some issues that still need to be worked out, but Colin is up front about them, and he’s motivated and eager to collaborate with people to solve them. He’s truly one of the good guys of free software and I enjoyed this talk a lot.

I also attended Ondrej Hudlicky’s session on software usability, which was entertaining but also thought-provoking. A lot of what goes into making good software we either take for granted or completely miss. It’s so easy for software to suck when you don’t start by thinking about what the user is trying to do, and making that easy. Although the slides were quite dense, Ondrej did a great job explaining the concepts and why they were important.

I also attended sessions on DNF’s SAT solver, caught a bit on static analysis that went way over my head, and saw Richard Hughes’ session on GNOME Software. DevConf.cz is so packed with content, it’s impossible to see more than about half of what you’d like. There’s so much more content for Java folks, low-level network and hardware hackers, and kernel jockeys that it makes your head spin!

In the evening I went with a bunch of folks to get pizza at the hilariously named Pizzeria Al Capone down the street. The food was quite good, and the beer plentiful as we swapped stories and jokes. We had people from all over the globe at the table so it was a great night. Afterward we retired to the famous bowling bar in the basement of the Hotel Avanti. And of course, more beer and stories. I turned in rather late, around 1:00am, but in good shape for the next morning.

DevConf.cz day 2, Saturday.

Started out the day early again, with a 9:00am session on Cockpit. Cockpit is a new Linux server management user interface that beautifully fits the look and feel of modern desktops. It’s also has already grown a lot of capability including user and storage administration. This is a great way for us to break away from clunky and individually deprecating system-config-* tools. Instead we can move to a tool that’s more flexible, extensible, and network transparent for scalability.

Following was a talk by Russ Doty on security concerns in platform and application development. It was mainly general but made some good points about where threats usually come from (hint: not Igor the evil state-funded hacker).

Of course, no DevConf.cz event would be complete without a rapid-fire presentation from Lennart Poettering, and this year was no exception. Lennart covered kdbus, a new kernel implementation of IPC based on the excellent D-Bus. Kdbus is on its way into the kernel and will make Linux even slicker, starting with early boot and extending all the way to latest shutdown.

I also sat in on Ric Wheeler’s excellent presentation on Persistent Memory, which is next generation storage technology. Ric covered some of the challenges in supporting new types of storage in the Linux kernel, and the relative strengths and weaknesses of each.

Afterward, I went to lunch with Ralph Bean and Pierre-Yves Chibon from the Fedora Engineering team. With us were Patrick Uiterwijk and folks from Red Hat that work on infrastructure and tools for RHEL and JBoss engineers. We discussed some areas of potential collaboration, including a messaging bus for Red Hat Bugzilla. That could be an awesome new input for contributor data.

Then all the smart folks went off to find better broadband at the hotel to pore over some code together. Since I wouldn’t have been much help, I went back to the conference to catch Simo Sorce’s talk on Kerberos.

Following Simo, Dan Walsh talked about secure Linux containers. As always he was tremendously entertaining. Dan joked about how he’s been a big proponent of libvirt-sandbox for secure container support, but recently “got religion” about Docker. I hope this was taped because it was really informative. No wonder Dan’s consistently rated as a top speaker at the Red Hat Summit. (Note, you can still register for the event; I’ll be there in San Francisco too!)

Next Kyle McMartin talked about the pleasure and pitfalls of porting the Linux kernel to new architectures (hello, aarch64!). I admit a lot of this went over my head, but Kyle told some funny stories about stalking weird bugs in test suites exposed by porting. At least I think they were funny. Or rather, I think some people thought they were funny, since they were all laughing. I don’t understand kernel people, but they’re mostly lovable, and many of them have awesome beards.

Finally, I saw a talk on Arduino Yún. This model includes a small, embedded Linux computer that you can make do all sorts of cool things with the built-in sensors and other capabilities. The talk made me wish I had more spare time to spend on learning how to do hardware tinkering. Where’s my time machine?

I bowed out of the lightning talks (even though some of them looked awesome) so I could drop my bag at the hotel before the night party at Klub Fléda, a sort of warehouse-y bar/music club nearby the conference venue. With beer beckoning, it’s time to relax a bit with friends and colleagues!

Tomorrow there will be Fedora focused sessions, so I’m really looking forward to that. More later…

Most hearty congratulations.

Sandro Mathys reports on his blog that he was selected as the 2010 RHCE of the Year for Europe. I'm not surprised to see another active Fedora contributor selected for this honor, like John Rose in 2009 for North America and Jeroen van Meeuwen for Europe and Michael Yingbull for Canada in 2008.

RHCEs take a challenging, practical test to ensure they have a high degree of capability and performance. If you want to do well on that test (or any other for that matter), you practice. RHCEs know as they practice their skills on the Red Hat Enterprise Linux platform, they can develop new skills in emerging technologies on Fedora for use now and in the future. I suspect a lot of RHCEs use RHEL at work and Fedora on their desktop or at home in part for that reason. But more than just developing skills, the Fedora community allows anyone — not just an RHCE — to work with some of the brightest people in free and open source software, and contribute to what they use.

Our community Infrastructure team, for example, has quite a number of RHCE-certified people involved as you might expect. This team that puts exceptional IT service management principles to work every day, providing services for the Fedora community. They also develop frameworks and applications we use, and moreover, they provide them as 100% free and open source software. Anyone can not only use that software, but modify it for their own use and redistribute it as well. There are may other areas in Fedora in which we have RHCEs scattered throughout teams as well.

In each case of the previous RHCE winners from Fedora, someone who's discovered the value of Fedora has also stepped up to bring some knowledge and skill back into the community. Sandro, for instance, a long-time Fedora Ambassador, is heading up the team that will bring the Fedora Users and Developers Conference to Zurich this fall. I'm proud to add Sandro to this growing list of honored community members. Congratulations, sir!

Elections are open.

During the next week, Fedora contributors will vote for open seats on both the Fedora Project Board and the Fedora Engineering Steering Committee (FESCo). The polls are now open for both elections through the Fedora Election System. (Remember to login or you won't see the voting link for an election.) The elections will close on Wednesday, 2010-05-26 at UTC 23:59.

If you aren't familiar with how the system works, check out the Fedora Elections Guide. I also encourage Fedora community members to review the logs from our Town Hall meetings, where the candidates talked openly about their goals and viewpoints. Get informed, and then vote appropriately.

I thanked our election volunteers previously, but I also wanted to say a quick thanks to Mike McGrath and Mark Chappell on the Fedora Infrastructure team, who both ensured that the elections were set up properly and ready for business. Mark is actually working on application upgrades and better usability,which I hope we'll be able to use in the next regular election cycle. It's great to see a contributor jumping in to make a difference and improve the systems we use regularly.

Enough of my yakkin', whaddya say? Let's boogie. Go vote!

Quick links: (Board election)  (FESCo election)

Rocky road made smooth.

Thanks to hard work by people working on the Fedora Infrastructure team, we have a newer Transifex working on translate.fp.o. I wrote more about this in an earlier post, so I won’t endlessly repeat the whistles and cheers of a grateful (Fedora) nation here. But it’s now, while the Docs and L10n teams are hip-deep in translation of release notes and other written content, that we really appreciate having that new version online.

In Fedora 11 and 12 cycles, we had to do a very painstaking process involving manual runs of the gettext utilities to produce translation files that the old Transifex could understand and deal with. The newer Transifex 0.7.4, on the other hand, understands perfectly the way that our documentation tool Publican produces translation files. It’s a tremendous time saver.

Now for Fedora 13 we just merge some git branch content and update the translation (POT/PO) files, and Transifex handles everything else for us. It’s made the process so smooth this release that at the Docs meetings I keep scratching my head and thinking, “Where’s the danger? Where’s the fear and loathing? The sturm und drang?” No more rocky road!

Except for the ice cream of course. Mmm, ice cream.


According to the Statistics page on the wiki, last week we passed 1 million IP checkins for Fedora 12 systems!  This is roughly on par with where Fedora 11 was at the same time after its release, although it’s hard to discern the actual number of installations worldwide.

Although IP addresses are a convenient and anonymous way to gather these statistics, they’re not foolproof.  But given our past experience and analysis, which you can see in more detail in the section on yum check-ins, we are confident we’re significantly undercounting installations. There are millions of existing systems running Fedora 11 and other previous releases as well, although older systems are no longer receiving updates and we recommend that people try the latest and best free software available. And in a significant number of cases there are NATs and proxies that further impact this undercounting.

The above considerations influence me to be skeptical when I hear answers to the question “How many systems are running ‘Foo’?”. Is the claim supported with hard numbers? Are those numbers public and independently verifiable? As part of Fedora’s dedication to transparency, I definitely take those questions seriously.

We’re always trying to think of ways to improve our statistics gathering that continue that tradition of transparency, respect users’ privacy, and support the globally mirrored infrastructure that works so well (thanks Infrastructure team!). If you’ve got a suggestion that takes those factors into account, and you can help implement it, let me and other folks know through the advisory-board list.

Election extensions.

As seen here on the fedora-advisory-board list:

Although Mike McGrath and the Infrastructure team don’t expect the server relocation to affect our upcoming elections, we want to make sure the community’s ability to vote is not unnecessarily affected given the timing.

The original voting period was December 8-15, and the infrastructure move is occurring over the weekend of the 12th:



I talked this morning with Mike, John Rose, and Nigel Jones, and here’s the plan we arrived at:

  • The voting period will start on December 5 instead of December 8. This ensures that, no matter what unforeseen circumstances occur, there will be a full weekend of uninterrupted voting time available to all community members. All townhalls will be finished by that point, since none are planned for FUDCon at this time.
  • Although Mike and the Infrastructure team believe they can keep the elections open throughout the move, in the event of any substantial outage (8 hours or more), we will extend the end of voting by an additional day. For any additional day of outage, we’ll extend the end of voting by another additional day. Again, we aren’t expecting to have to do this, but it’s better for us to have a plan ahead of time. If there is no outage, voting will end on December 15 as originally planned.
  • Nigel is preparing a change to the elections app that will allow a logged in user to check the accuracy of a previously recorded vote. This will give an additional measure of confidence in the election system, even in the event of a brief outage. If any user detects a problem, they can report it via IRC, email to the infrastructure list, or a Trac ticket.

We’ll be putting announcements out elsewhere as well — but as Darren noted, there’s nothing wrong with getting your vote in early!

Fedora Talk activity day.

For a while now, I’ve been working on plans for a FAD to work on our Fedora Talk VoIP system. There are a couple gaps I’d really like to fill, such as the ability to record, publish, and/or stream calls. These features would make the system much more capable of high transparency. They’d also contribute to an archival history that might help future contributors in the same way as other types of conference and meeting proceedings.

For example, it would be really cool if we could use the Talk server to record our sessions at FUDCon, which could be streamed and/or downloaded later by any community member. The only thing we’d need to do is to run a microphone into any laptop running a VoIP softphone on Fedora Talk, set the record level, and voila.

The raison d’être for any Fedora Activity Day is to gather a small number of interested contributors to a central location to work on short-term, focused goals that advance some part of the Fedora Project. (These are very different from the Fedora Users and Developers Conferences (FUDCons), which are much larger gatherings that serve many needs.) I’ve been to a couple of FADs this year already, including one for Documentation in Clemson, SC, and another for the development cycle in Raleigh, NC. So, in keeping with the slow sojourn up the East Coast, I am working on one that will be held in my hometown of Fredericksburg, VA.

It might be nice to add one more person with some Python and Infrastructure skills to our merry band, especially if they know something about streaming media. If you’re interested, drop me a line and put your name on the wiki page.

If I’m lyin’, I’m dyin’.

Some important statistics from the first week of Fedora 11 release:

  • Over 140 Terabytes of Fedora 11 shipped via BitTorrent.
  • Approximately 200,000 direct downloads from unique IP addresses. (Incidentally, there were over 600,000 requests but some IP addresses requested more than one download.)
  • Over 1,000,000 1,200,000 visits to our web and wiki site in just seven days.

Note that our expert Fedora Infrastructure team made all this traffic almost unnoticeable to people, instead of crushing our servers to their knees. Hopefully Mike McGrath and some of the other team members will post a little bit about how they pull all this off. (Hint, hint!) I know that we use memcached, and that MirrorManager, maintained by Matt Domsch, figures heavily into our ability to get people to the closest Fedora bits when they request a download.

It never ceases to amaze me that our releases don’t seem to cause meltdowns like they used to. I think the Infrastructure team secretly yearns for release days to be more exciting, but it’s ironic that their own success makes that less likely. 😉