Long live Release Engineering

My involvement in Fedora goes back to late 2003 early 2004 somewhere as a packager for fedora.us. I started by getting a few packages in to scratch some of my itches and I saw it as a way to give back to the greater open source community. Around FC3 somewhere I stepped up to help in infrastructure to rebuild the builders in plague, the build system we used before koji and that we used for EPEL(Something that I helped form) for awhile until we got external repo support in koji.

I was involved in the implementation of koji in Fedora, I joined OLPC as a build and release engineer, where I oversaw a move of the OS they shipped from FC6 to F8, and laid a foundation for the move to F9. I left OLPC when Red Hat opensourced RHN Satellite as “spacewalk project” I joined Red Hat as the release engineer for both, after a brief period there was some reorganisation in engineering that resulted in me handing off the release engineering tasks to someone closer the the engineers working on the code. As a result I worked on Fedora full time helping Jesse Keating. When he decided to work on the internal migration from CVS to git I took over as the lead.

During Fedora 14 the transition was made from Jesse to me. For the following 10 Fedora releases I was the primary person doing the work to get Fedora out the door. During that time there has been tremendous change in Fedora, how we do things and what we deliver. Fedora 14 shipped with 12905 packages, 1 install tree, a handful of livecd’s across two architectures. In Fedora 24 we shipped with 19760 packages, 4 install tree’s, 10 livecd’s, Cloud Images, Vagrant Box images, Container Base images, Atomic Host (Iamges and ostree) across 3 architectures. Along with all the primary deliverables we had a much more robust and functional Alternative Architecture program running. In Fedora 26 we added aarch64 and ppc64le to primary koji and in Fedora 27 we added s390x.

During this time we added things like Fedora Editions, and support for many new technologies. The tooling we used to compose Fedora grew in complexity to meet the growing demands and reduce the need for people to manually do the work. Management of the development of the tooling we use was taken over by different teams inside of Red Hat working upstream in the community.  Fedora Release Engineering gained a project manager who helped us to grow and become less of a black box and deal with the growing pains we faced. Fedora Infrastructure provided people to help develop and deploy the ability to build layered images for containers. Pungi the compose tool got a major version bump and grew up a lot. We also developed and worked with upstream koji to get new features and functionality into the buildsystem. Release Engineering was one of  the first adopters of pagure.

Mohan Boddu Joined Red Hat to help with Fedora just before Fedora 25, we worked on Fedora 25 together and Mohan has primarily been the person responsible for composing and making sure we ship Fedora since. During that time I have been the Team Lead for the platform team in release engineering inside of Red Hat, I have been spending my time between Fedora and RHEL and making sure that we bring together the way we build and compose both Operating Systems.

Recently I have accepted a Job offer to become the manager of a different team inside of Red Hat. My new role will be working on multi-arch support for internal products. As a result of my change in roles inside of Red Hat I will be stepping down on Friday the 23rd of March as the lead for release engineering in Fedora so I can focus on my new role. Mohan will be taking over for me,  he will be helped by the current project manager Suzanne Yeghiayan along with a cast of a handful who work tirelessly to ensure we can ship everything. All requests for projects should go though pagure or taiga for grooming and prioritisation.

Please give Mohan a big congratulations and be sure to make sure that you work with Suzanne to get your requests prioritised.

What is going on in Fedora Release Engineering

As we rapidly hit the last days of Fedora 26 development, the final notches of planning for what Release Engineering can get done in Fedora 27 is coming together. Our priorities are mostly sorted, and we are marking things to target for Fedora 28.

Release Engineering has its backlog in a Taiga instance that is hosted in the fedora infrastructure cloud. As you can see the backlog is very full. We have a limited number of people Mohan, Adam, Randy, Patrick, and Rob have very full plates with day to day work and getting new features done as possible, Kevin and myself do what we can to assist. If there is something you really want to see working with us to get it scoped and getting it implemented ensures the best chance of success.

My role within releng has changed over the last few months, Fedora is no longer my only responsibility, I am now the team lead for Platform Release Engineering inside of Red Hat, I oversee Fedora and RHEL as a result I am less hands on  in the day to day doing things. I ran one compose in Fedora 25 and so far in Fedora 26 I have yet to run a production compose.  As a result if you ping me directly asking for input I will redirect you to file an issue in pagure, or to mail the releng mailing list, this is not because I do not want to help, but because I want to remove myself as a single point of failure in getting things done. Of course if you feel you are not getting a responsive or appropriate answer please ping and engage with me so I can help resolving the issue.




Asking for help with koji builds

People often ask for help in #fedora-releng or #fedora-devel with koji builds. The number 1 mistake people make is to post links to log files. The problem with posting to logs is that quite often the log file people point at is the wrong one.  It helps in debugging when we have full access to all log files ad the parent and other children tasks in koji.

The easiest path to providing assistance is to point at the task in question, sometimes we have to look at other arches and other log files in order to figure out what exactly has gone wrong. So please in order to help you better post links to tasks and not log files.

Thank you

installing openh264 on Fedora 24

Lately we have been working with the Workstation Working Group to setup the ability to build and make available a legal h.264 codec for fedora. Cisco has made available a patent grant for the codec,  you can find out more about it at the openh264 project.  One of the main conditions is that you have to download the binary rpms from Cisco directly. In order to meet this requirement we have blocked downloading the built rpms from koji, and put up a site hosting the repodata. The download of the rpms then comes from Cisco.


if you are using gnome and you go to play media using h.264 you should be prompted to enable the repo and install the rpms.  If you are using any other environment or want to just use dnf, follow the following steps:

Enable the repo, it ships disabled with metadata enabled

$ sudo dnf config-manager --set-enabled fedora-cisco-openh264

then install the mozilla integration and the gstreamer plugin

$ sudo dnf install mozilla-openh264 gstreamer1-plugin-openh264
Importing GPG key 0x81B46521:
 Userid : "Fedora (24) <fedora-24-primary@fedoraproject.org>"
 Fingerprint: 5048 BDBB A5E7 76E5 47B0 9CCC 73BD E983 81B4 6521
 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-armhfp
Is this ok [y/N]: y
Fedora 24 openh264 (From Cisco) - armhfp 
Last metadata expiration check: 0:02:13 ago on Wed May 11 13:38:09 2016.
Dependencies resolved.
 Package Arch Version Repository Size
 gstreamer1-plugin-openh264 armv7hl 1.6.1-2.fc24 fedora-cisco-openh264 20 k
 mozilla-openh264 armv7hl 1.5.2-0.4.git21e44bd.fc24 fedora-cisco-openh264 295 k
 openh264 armv7hl 1.5.2-0.4.git21e44bd.fc24 fedora-cisco-openh264 290 k
Transaction Summary
Install 3 Packages
Total download size: 604 k
Installed size: 1.4 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): gstreamer1-plugin-openh264-1.6.1-2.fc24.armv7hl.rpm 12 kB/s | 20 kB 00:01 
(2/3): mozilla-openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl.rpm 168 kB/s | 295 kB 00:01 
(3/3): openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl.rpm 158 kB/s | 290 kB 00:01 
Total 322 kB/s | 604 kB 00:01 
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
 Installing : openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl 1/3 
 Installing : mozilla-openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl 2/3 
 Installing : gstreamer1-plugin-openh264-1.6.1-2.fc24.armv7hl 3/3 
 Verifying : mozilla-openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl 1/3 
 Verifying : gstreamer1-plugin-openh264-1.6.1-2.fc24.armv7hl 2/3 
 Verifying : openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl 3/3
 gstreamer1-plugin-openh264.armv7hl 1.6.1-2.fc24 mozilla-openh264.armv7hl 1.5.2-0.4.git21e44bd.fc24 openh264.armv7hl 1.5.2-0.4.git21e44bd.fc24

The repodata for this repo and the rpms are signed with the fedora key. right now we only have the fedora 24 repo up, we will be aiming to finalise the process and get Rawhide and Fedora 23 rpms and repos up this week also.

I would  like to thank the Workstation Working group and Cisco for sorting out the agreements for this to be possible.

Working in the new Release Engineering world

There is a new position opened up in Red Hat for someone to help in Release Engineering,  you can find the job details here.  The role will be working inside of Red Hat as well as in Fedora. It is based in Westford, MA, however there are some more jobs opening up soon that are in other parts of the world. I will be posting updates as the other Jobs open.  If you are interested and have further questions contact me and ask, or just go and apply.

Fedora Releng demo

In Fedora Release Engineering land we have been undertaking some big changes in how we work.  Most of these changes have been quieter than they should have been.  We had the latest demo of our work today, you can watch the video on youtube.

We are using an instance of taiga hosted in fedora space to be able to work smarter and more efficiently, as well as to enable more visibility into what we are doing and what we have planned. The ultimate goal is to make it easy for anyone to know what we are doing, and to contribute as they can. The contribution may be writing code, or just helping to make sure that requirements are fully documented.

I intend to take some time and write about the different things we are working on and help to generally make our work more visible to everyone. If you have any questions you want answered by releng you could ask in a comment here,  but better would be to ask on the mailing list or in #fedora-releng on freenode


FUDCon LATAM started this morning in Managua Nicaragua. The turnout is great and there is a high level of anticipation of what is to come in the next few days.  The organisers have done a really good job in planning. I for one am looking forward to seeing what is new in the Fedora world in LATAM.

The LATAM Fedora community is very passionate about Fedora and growing every day.  I will be writing more about the happenings over the next few days

Great turnout for FUDCon Latam


Underway is FUDCon LATAM

The 2013 edition of FUDCon LATAM is underway, the host Cusco, Peru has done a fantastic job organising the event. There had to be at least 500 people at the opening event. There was 65 talks proposed, with the barcamp voting being an interesting experience. Cusco the host city is beautiful, the part of the city where FUDCon is help is really old and well cared for. there is big old Churchs, open green spaces, and some great architecture.

huge crowd in Peru for FUDCon LATAM