Jobs Red Hat

Multi-Arch jobs

In the time since I left Release Engineering I have been managing the team inside of Red Hat that is working on enabling OpenShift and OpenStack on non Intel architectures. We have delivered OpenStack 13 and 14 for Power, as well as Openshift 3.10 and 3.11 for Power. We are currently working on the next releases of Openstack and Openshift, as well as Multi-Arch support for CoreOS both in Fedora and internally at Red Hat. As a result we are expanding the team and currently have 4 positions available. The Jobs are listed as being a Principal level in Boston, they are however open to all levels up to Principal and can be remote.


OpenShift and CoreOS


Long live Release Engineering

My involvement in Fedora goes back to late 2003 early 2004 somewhere as a packager for I started by getting a few packages in to scratch some of my itches and I saw it as a way to give back to the greater open source community. Around FC3 somewhere I stepped up to help in infrastructure to rebuild the builders in plague, the build system we used before koji and that we used for EPEL(Something that I helped form) for awhile until we got external repo support in koji.

I was involved in the implementation of koji in Fedora, I joined OLPC as a build and release engineer, where I oversaw a move of the OS they shipped from FC6 to F8, and laid a foundation for the move to F9. I left OLPC when Red Hat opensourced RHN Satellite as “spacewalk project” I joined Red Hat as the release engineer for both, after a brief period there was some reorganisation in engineering that resulted in me handing off the release engineering tasks to someone closer the the engineers working on the code. As a result I worked on Fedora full time helping Jesse Keating. When he decided to work on the internal migration from CVS to git I took over as the lead.

During Fedora 14 the transition was made from Jesse to me. For the following 10 Fedora releases I was the primary person doing the work to get Fedora out the door. During that time there has been tremendous change in Fedora, how we do things and what we deliver. Fedora 14 shipped with 12905 packages, 1 install tree, a handful of livecd’s across two architectures. In Fedora 24 we shipped with 19760 packages, 4 install tree’s, 10 livecd’s, Cloud Images, Vagrant Box images, Container Base images, Atomic Host (Iamges and ostree) across 3 architectures. Along with all the primary deliverables we had a much more robust and functional Alternative Architecture program running. In Fedora 26 we added aarch64 and ppc64le to primary koji and in Fedora 27 we added s390x.

During this time we added things like Fedora Editions, and support for many new technologies. The tooling we used to compose Fedora grew in complexity to meet the growing demands and reduce the need for people to manually do the work. Management of the development of the tooling we use was taken over by different teams inside of Red Hat working upstream in the community.  Fedora Release Engineering gained a project manager who helped us to grow and become less of a black box and deal with the growing pains we faced. Fedora Infrastructure provided people to help develop and deploy the ability to build layered images for containers. Pungi the compose tool got a major version bump and grew up a lot. We also developed and worked with upstream koji to get new features and functionality into the buildsystem. Release Engineering was one of  the first adopters of pagure.

Mohan Boddu Joined Red Hat to help with Fedora just before Fedora 25, we worked on Fedora 25 together and Mohan has primarily been the person responsible for composing and making sure we ship Fedora since. During that time I have been the Team Lead for the platform team in release engineering inside of Red Hat, I have been spending my time between Fedora and RHEL and making sure that we bring together the way we build and compose both Operating Systems.

Recently I have accepted a Job offer to become the manager of a different team inside of Red Hat. My new role will be working on multi-arch support for internal products. As a result of my change in roles inside of Red Hat I will be stepping down on Friday the 23rd of March as the lead for release engineering in Fedora so I can focus on my new role. Mohan will be taking over for me,  he will be helped by the current project manager Suzanne Yeghiayan along with a cast of a handful who work tirelessly to ensure we can ship everything. All requests for projects should go though pagure or taiga for grooming and prioritisation.

Please give Mohan a big congratulations and be sure to make sure that you work with Suzanne to get your requests prioritised.

27 Fedora releng whatcanido

What is going on in Fedora Release Engineering

As we rapidly hit the last days of Fedora 26 development, the final notches of planning for what Release Engineering can get done in Fedora 27 is coming together. Our priorities are mostly sorted, and we are marking things to target for Fedora 28.

Release Engineering has its backlog in a Taiga instance that is hosted in the fedora infrastructure cloud. As you can see the backlog is very full. We have a limited number of people Mohan, Adam, Randy, Patrick, and Rob have very full plates with day to day work and getting new features done as possible, Kevin and myself do what we can to assist. If there is something you really want to see working with us to get it scoped and getting it implemented ensures the best chance of success.

My role within releng has changed over the last few months, Fedora is no longer my only responsibility, I am now the team lead for Platform Release Engineering inside of Red Hat, I oversee Fedora and RHEL as a result I am less hands on  in the day to day doing things. I ran one compose in Fedora 25 and so far in Fedora 26 I have yet to run a production compose.  As a result if you ping me directly asking for input I will redirect you to file an issue in pagure, or to mail the releng mailing list, this is not because I do not want to help, but because I want to remove myself as a single point of failure in getting things done. Of course if you feel you are not getting a responsive or appropriate answer please ping and engage with me so I can help resolving the issue.




Fedora koji releng

Asking for help with koji builds

People often ask for help in #fedora-releng or #fedora-devel with koji builds. The number 1 mistake people make is to post links to log files. The problem with posting to logs is that quite often the log file people point at is the wrong one.  It helps in debugging when we have full access to all log files ad the parent and other children tasks in koji.

The easiest path to providing assistance is to point at the task in question, sometimes we have to look at other arches and other log files in order to figure out what exactly has gone wrong. So please in order to help you better post links to tasks and not log files.

Thank you

23 24 Fedora h.264

installing openh264 on Fedora 24

Lately we have been working with the Workstation Working Group to setup the ability to build and make available a legal h.264 codec for fedora. Cisco has made available a patent grant for the codec,  you can find out more about it at the openh264 project.  One of the main conditions is that you have to download the binary rpms from Cisco directly. In order to meet this requirement we have blocked downloading the built rpms from koji, and put up a site hosting the repodata. The download of the rpms then comes from Cisco.


if you are using gnome and you go to play media using h.264 you should be prompted to enable the repo and install the rpms.  If you are using any other environment or want to just use dnf, follow the following steps:

Enable the repo, it ships disabled with metadata enabled

$ sudo dnf config-manager --set-enabled fedora-cisco-openh264

then install the mozilla integration and the gstreamer plugin

$ sudo dnf install mozilla-openh264 gstreamer1-plugin-openh264
Importing GPG key 0x81B46521:
 Userid : "Fedora (24) <>"
 Fingerprint: 5048 BDBB A5E7 76E5 47B0 9CCC 73BD E983 81B4 6521
 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-24-armhfp
Is this ok [y/N]: y
Fedora 24 openh264 (From Cisco) - armhfp 
Last metadata expiration check: 0:02:13 ago on Wed May 11 13:38:09 2016.
Dependencies resolved.
 Package Arch Version Repository Size
 gstreamer1-plugin-openh264 armv7hl 1.6.1-2.fc24 fedora-cisco-openh264 20 k
 mozilla-openh264 armv7hl 1.5.2-0.4.git21e44bd.fc24 fedora-cisco-openh264 295 k
 openh264 armv7hl 1.5.2-0.4.git21e44bd.fc24 fedora-cisco-openh264 290 k
Transaction Summary
Install 3 Packages
Total download size: 604 k
Installed size: 1.4 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): gstreamer1-plugin-openh264-1.6.1-2.fc24.armv7hl.rpm 12 kB/s | 20 kB 00:01 
(2/3): mozilla-openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl.rpm 168 kB/s | 295 kB 00:01 
(3/3): openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl.rpm 158 kB/s | 290 kB 00:01 
Total 322 kB/s | 604 kB 00:01 
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
 Installing : openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl 1/3 
 Installing : mozilla-openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl 2/3 
 Installing : gstreamer1-plugin-openh264-1.6.1-2.fc24.armv7hl 3/3 
 Verifying : mozilla-openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl 1/3 
 Verifying : gstreamer1-plugin-openh264-1.6.1-2.fc24.armv7hl 2/3 
 Verifying : openh264-1.5.2-0.4.git21e44bd.fc24.armv7hl 3/3
 gstreamer1-plugin-openh264.armv7hl 1.6.1-2.fc24 mozilla-openh264.armv7hl 1.5.2-0.4.git21e44bd.fc24 openh264.armv7hl 1.5.2-0.4.git21e44bd.fc24

The repodata for this repo and the rpms are signed with the fedora key. right now we only have the fedora 24 repo up, we will be aiming to finalise the process and get Rawhide and Fedora 23 rpms and repos up this week also.

I would  like to thank the Workstation Working group and Cisco for sorting out the agreements for this to be possible.


Want to work in Release Engineering in Europe?

Red Hat Release Engineering is hiring in Europe. The job is primarily internal, however we are working to align Fedora and internal tooling and processes, so there will be a lot of overlap.  The job info can be found here. If you have more questions or want to apply contact me to get things started.


Working in the new Release Engineering world

There is a new position opened up in Red Hat for someone to help in Release Engineering,  you can find the job details here.  The role will be working inside of Red Hat as well as in Fedora. It is based in Westford, MA, however there are some more jobs opening up soon that are in other parts of the world. I will be posting updates as the other Jobs open.  If you are interested and have further questions contact me and ask, or just go and apply.


Fedora Releng demo

In Fedora Release Engineering land we have been undertaking some big changes in how we work.  Most of these changes have been quieter than they should have been.  We had the latest demo of our work today, you can watch the video on youtube.

We are using an instance of taiga hosted in fedora space to be able to work smarter and more efficiently, as well as to enable more visibility into what we are doing and what we have planned. The ultimate goal is to make it easy for anyone to know what we are doing, and to contribute as they can. The contribution may be writing code, or just helping to make sure that requirements are fully documented.

I intend to take some time and write about the different things we are working on and help to generally make our work more visible to everyone. If you have any questions you want answered by releng you could ask in a comment here,  but better would be to ask on the mailing list or in #fedora-releng on freenode


Fedora atomic job opening

Want to work on fedora in new and interesting ways? Red Hat is hiring a person to work on atomic in fedora. Full details can be found here, however the short summary is that you would work with me and the rest of the fedora releng team along with infrastructure on building new and better ways to deliver fedora with a focus on atomic and related technologies.



FUDCon LATAM started this morning in Managua Nicaragua. The turnout is great and there is a high level of anticipation of what is to come in the next few days.  The organisers have done a really good job in planning. I for one am looking forward to seeing what is new in the Fedora world in LATAM.

The LATAM Fedora community is very passionate about Fedora and growing every day.  I will be writing more about the happenings over the next few days