Long live Release Engineering

My involvement in Fedora goes back to late 2003 early 2004 somewhere as a packager for fedora.us. I started by getting a few packages in to scratch some of my itches and I saw it as a way to give back to the greater open source community. Around FC3 somewhere I stepped up to help in infrastructure to rebuild the builders in plague, the build system we used before koji and that we used for EPEL(Something that I helped form) for awhile until we got external repo support in koji.

I was involved in the implementation of koji in Fedora, I joined OLPC as a build and release engineer, where I oversaw a move of the OS they shipped from FC6 to F8, and laid a foundation for the move to F9. I left OLPC when Red Hat opensourced RHN Satellite as “spacewalk project” I joined Red Hat as the release engineer for both, after a brief period there was some reorganisation in engineering that resulted in me handing off the release engineering tasks to someone closer the the engineers working on the code. As a result I worked on Fedora full time helping Jesse Keating. When he decided to work on the internal migration from CVS to git I took over as the lead.

During Fedora 14 the transition was made from Jesse to me. For the following 10 Fedora releases I was the primary person doing the work to get Fedora out the door. During that time there has been tremendous change in Fedora, how we do things and what we deliver. Fedora 14 shipped with 12905 packages, 1 install tree, a handful of livecd’s across two architectures. In Fedora 24 we shipped with 19760 packages, 4 install tree’s, 10 livecd’s, Cloud Images, Vagrant Box images, Container Base images, Atomic Host (Iamges and ostree) across 3 architectures. Along with all the primary deliverables we had a much more robust and functional Alternative Architecture program running. In Fedora 26 we added aarch64 and ppc64le to primary koji and in Fedora 27 we added s390x.

During this time we added things like Fedora Editions, and support for many new technologies. The tooling we used to compose Fedora grew in complexity to meet the growing demands and reduce the need for people to manually do the work. Management of the development of the tooling we use was taken over by different teams inside of Red Hat working upstream in the community.  Fedora Release Engineering gained a project manager who helped us to grow and become less of a black box and deal with the growing pains we faced. Fedora Infrastructure provided people to help develop and deploy the ability to build layered images for containers. Pungi the compose tool got a major version bump and grew up a lot. We also developed and worked with upstream koji to get new features and functionality into the buildsystem. Release Engineering was one of  the first adopters of pagure.

Mohan Boddu Joined Red Hat to help with Fedora just before Fedora 25, we worked on Fedora 25 together and Mohan has primarily been the person responsible for composing and making sure we ship Fedora since. During that time I have been the Team Lead for the platform team in release engineering inside of Red Hat, I have been spending my time between Fedora and RHEL and making sure that we bring together the way we build and compose both Operating Systems.

Recently I have accepted a Job offer to become the manager of a different team inside of Red Hat. My new role will be working on multi-arch support for internal products. As a result of my change in roles inside of Red Hat I will be stepping down on Friday the 23rd of March as the lead for release engineering in Fedora so I can focus on my new role. Mohan will be taking over for me,  he will be helped by the current project manager Suzanne Yeghiayan along with a cast of a handful who work tirelessly to ensure we can ship everything. All requests for projects should go though pagure or taiga for grooming and prioritisation.

Please give Mohan a big congratulations and be sure to make sure that you work with Suzanne to get your requests prioritised.

What is going on in Fedora Release Engineering

As we rapidly hit the last days of Fedora 26 development, the final notches of planning for what Release Engineering can get done in Fedora 27 is coming together. Our priorities are mostly sorted, and we are marking things to target for Fedora 28.

Release Engineering has its backlog in a Taiga instance that is hosted in the fedora infrastructure cloud. As you can see the backlog is very full. We have a limited number of people Mohan, Adam, Randy, Patrick, and Rob have very full plates with day to day work and getting new features done as possible, Kevin and myself do what we can to assist. If there is something you really want to see working with us to get it scoped and getting it implemented ensures the best chance of success.

My role within releng has changed over the last few months, Fedora is no longer my only responsibility, I am now the team lead for Platform Release Engineering inside of Red Hat, I oversee Fedora and RHEL as a result I am less hands on  in the day to day doing things. I ran one compose in Fedora 25 and so far in Fedora 26 I have yet to run a production compose.  As a result if you ping me directly asking for input I will redirect you to file an issue in pagure, or to mail the releng mailing list, this is not because I do not want to help, but because I want to remove myself as a single point of failure in getting things done. Of course if you feel you are not getting a responsive or appropriate answer please ping and engage with me so I can help resolving the issue.




Asking for help with koji builds

People often ask for help in #fedora-releng or #fedora-devel with koji builds. The number 1 mistake people make is to post links to log files. The problem with posting to logs is that quite often the log file people point at is the wrong one.  It helps in debugging when we have full access to all log files ad the parent and other children tasks in koji.

The easiest path to providing assistance is to point at the task in question, sometimes we have to look at other arches and other log files in order to figure out what exactly has gone wrong. So please in order to help you better post links to tasks and not log files.

Thank you

installing openh264 on Fedora 24

Lately we have been working with the Workstation Working Group to setup the ability to build and make available a legal h.264 codec for fedora. Cisco has made available a patent grant for the codec,  you can find out more about it at the openh264 project.  One of the main conditions is that you have to download the binary rpms from Cisco directly. In order to meet this requirement we have blocked downloading the built rpms from koji, and put up a site hosting the repodata. The download of the rpms then comes from Cisco.


if you are using gnome and you go to play media using h.264 you should be prompted to enable the repo and install the rpms.  If you are using any other environment or want to just use dnf, follow the following steps:

Enable the repo, it ships disabled with metadata enabled

$ sudo dnf config-manager --set-enabled fedora-cisco-openh264

then install the mozilla integration and the gstreamer plugin

$ sudo dnf install mozilla-openh264 gstreamer1-plugin-openh264
Importing GPG key 0x81B46521:
 Userid : "Fedora (24) <fedora-24-primary@fedoraproject.org>"
 Fingerprint: 5048 BDBB A5E7 76E5 47B0 9CCC 73BD E983 81B4 6521
 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-armhfp
Is this ok [y/N]: y
Fedora 24 openh264 (From Cisco) - armhfp 
Last metadata expiration check: 0:02:13 ago on Wed May 11 13:38:09 2016.
Dependencies resolved.
 Package Arch Version Repository Size
 gstreamer1-plugin-openh264 armv7hl 1.6.1-2.fc24 fedora-cisco-openh264 20 k
 mozilla-openh264 armv7hl 1.5.2-0.4.git21e44bd.fc24 fedora-cisco-openh264 295 k
 openh264 armv7hl 1.5.2-0.4.git21e44bd.fc24 fedora-cisco-openh264 290 k
Transaction Summary
Install 3 Packages
Total download size: 604 k
Installed size: 1.4 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): gstreamer1-plugin-openh264-1.6.1-2.fc24.armv7hl.rpm 12 kB/s | 20 kB 00:01 
(2/3): mozilla-openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl.rpm 168 kB/s | 295 kB 00:01 
(3/3): openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl.rpm 158 kB/s | 290 kB 00:01 
Total 322 kB/s | 604 kB 00:01 
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
 Installing : openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl 1/3 
 Installing : mozilla-openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl 2/3 
 Installing : gstreamer1-plugin-openh264-1.6.1-2.fc24.armv7hl 3/3 
 Verifying : mozilla-openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl 1/3 
 Verifying : gstreamer1-plugin-openh264-1.6.1-2.fc24.armv7hl 2/3 
 Verifying : openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl 3/3
 gstreamer1-plugin-openh264.armv7hl 1.6.1-2.fc24 mozilla-openh264.armv7hl 1.5.2-0.4.git21e44bd.fc24 openh264.armv7hl 1.5.2-0.4.git21e44bd.fc24

The repodata for this repo and the rpms are signed with the fedora key. right now we only have the fedora 24 repo up, we will be aiming to finalise the process and get Rawhide and Fedora 23 rpms and repos up this week also.

I would  like to thank the Workstation Working group and Cisco for sorting out the agreements for this to be possible.