Red Hat Summit is just a few weeks away, and it’s a great place to learn about RDO, and deploying OpenStack on Red Hat Enterprise Linux or CentOS.

If you’re coming, be sure to drop by the RDO booth in the Community Central section of the exhibit hall. Grab an RDO tshirt, watch demos of what people are doing on top of RDO, and we’ll have experts there to answer your questions.

You’ll want to mark these sessions on your agenda. Get there early, as sessions often fill up pretty quick.

• Peanut butter and jelly: Mapping the deep integration between OpenStack and Ceph - Federico Lucifredi, Sebastien Han, Sean Cohen

• Best practices for deploying and scaling Red Hat OpenStack Platform with Open vSwitch and Red Hat Ceph storage - Jacob Liberman

• OpenStack-as-a-Service: Red Hat and Rackspace - Manuel Rodriguez, Nicholas Gerasimatos, Jeff Ekstrom

• Become an OpenStack TripleO ATC, easy as ABC - K. Rain Leander

• Running a policy-based cloud with Cisco Application Centric Infrastructure, Red Hat OpenStack, and Project Contiv - Mike Cohen

• Hot topics in containers, OpenStack and Hadoop - Duane DeCapite, Krishna Mayuram

• A container stack for OpenStack - Stephen Gordon, Sebastien Han, Scott McCarty

• Influencing OpenStack documentation - Christopher Wawak

• Software-defined networking (SDN) fundamentals for NFV, OpenStack, and containers - Nir Yechiel, Rashid Khan

• Hands-on deep dive with Red Hat Enterprise Linux OpenStack Platform - Guilherme Barros, Rhys Oxenham

• Red Hat Enterprise Linux OpenStack Platform NFV solution overview and roadmap - Kyle Forster, Sandro Mazziotta

• Successfully building a private cloud with OpenStack - Soumen Saha, Richard Haigh, Margaret Dawson

• When flexibility met simplicity: The friendship of OpenStack and Ansible - Major Hayden, Robyn Bergeron

• Repeatable, reliable OpenStack deployments: Pipe dream or reality? - Randy Perryman

• MySQL, containers, and Ceph - Yves Trudeau, Kyle Bader

• Use Linux on your whole rack with RDO and open networking - Scott Suehle

• Deploying an Overcloud with Red Hat Enterprise Linux OpenStack Platform Director

• OpenStack instance management using the command-line interface

Time goes fast as a RDO newbie and once I understood what the RDO Trunk and CloudSIG repos provide (read my previous post if you still have doubts), my next goal was to have a clear high-level perspective of the entire delivery workflow followed in the project.

Although there is a lot of detailed and great information in different sources (see More info at the end of the post), it may be useful for others to provide a high level and global description of it. In this post I’ll focus in the process around RDO Trunk repositories as shown in the following diagram:

Code repositories

Upstream Code

As for any packaging project, everything starts with the upstream code. For RDO, it is provided by OpenStack projects in https://github.com/openstack/<project> . The process to manage these repos is defined by OpenStack and can be found in their Developer Guide and the Release Process.

In RDO, these repositories are used to fetch the code that will be packaged for both RDO Trunk and CloudSIG in different ways:

• For RDO Trunk, the automatic packager, DLRN clones the involved git repo.
• For RDO CloudSIG, the specific tarball for the point release to be packaged will be fetched at packaging time.

Distgit repo

As described in the great RDO packaging guide, ditgit is the git repository “where the .spec file lives” . For RDO, distgit repos are located in https://review.rdoproject.org/r/#/admin/projects/openstack/<project>-distgit (you can see the entire list here). Typically they contain the spec files used to build RPM packages, systemd unit files and some system config files (i.e. logrotate or sudoers configuration).

Because of the reproducibility requirements of RDO CloudSIG packaging, spec files for these and trunk repos differ slightly:

• Changelog, version and release fields for the trunk is automatically generated while it must be manually created for stable packages.
• The source code is fetched from a versioned tarball for stable.

Also, we usually have different requirements for each OpenStack version so different branches are created based on version and RDO “flavor” (trunk or CloudSIG):

• rpm-<version> : is used to package RDO trunk (version can be master, mitaka or liberty )
• <version>-rdo : is used for RDO CloudSIG (as this is only for released version, master-rdo does not exist)

To ensure proper management of distgit repos, required changes follow a review process in RDO rpmfactory quite similar to the OpenStack project. Before merging the change in the actual repository, every request (review) will trigger a test job in the CI system that will post its -/+1 vote depending on the results and will require a core contributor to review and validate it.

DLRN: automatic packager for RDO Trunk

Packaging for RDO trunk is done by DLRN, a tool to build RPM packages on each commit merged in a set of configurable repositories. DLRN uses rdopkg to retrieve the metadata and repositories associated with each package (code and distgit) and mock to carry out the actual build.

NVR for packages generated by DLRN follows some rules:

• Version is set to MAJOR.MINOR.PATCH of the next upstream version.
• Release is 0.<timestamp>.<short commit hash>

For example, python-novaclient-3.4.1-0.20160412103538.77f214b.el7.centos.noarch.rpm is a package containing code released after GA of 3.4.0 and before 3.4.1 containing code until upstream commit 77f214b.

Once the package has been successfully built, DLRN creates a yum repository containing the latest version of every package successfully built at that time. This repo will not change and is created in a unique directory based on the specific commit hash. These repos will be available if a user wants to test an installation based on a specific commit. Finally, DLRN manages two “shorcuts” for repositories:

• current: will point to the last created per-commit repo.
• consistent: current repo, may not contain the latest versions of packages that failed to build (i.e. let’s assume that packaging of python-nova package failed while a later build of python-cinder succeed, in this case, current would contain the latest upstream version of cinder but not for nova). To avoid this situation, a new pointer “consistent” is managed by DLRN and will point to the last snapshot that contained package versions in sync with upstream versions.

Also, DLRN creates some html reports that can be used to check the results of each single build and provides the logs for troubleshooting any error while building the package.

Continuous Integration in RDO Trunk

To ensure the packages built by DLRN work as expected, the RDO project manages some CI jobs pipeline in the CentOS CI system. These jobs are created using jenkins-job-builder with the configuration managed in rdo-infra github. These jobs deploy and test RDO in different scenarios (packstack, tripleo quickstart and puppet) based mostly on WeIRDO (take a look into weirdo documentation) using the consistent repository. If all tests in the pipeline success, the content of the tested repo will be promoted as current-passed-ci. In this way, this link always points to the latest consistent repo that passed the CI jobs.

Additionally, current-tripleo repo is managed by a set of jobs focused on testing TripleO installer on the current-passed-ci repo. When those tests pass, this repo is promoted by this CI pipeline.

Access to repositories

All the repositories and content managed by DLRN and CI jobs are directly accesible for public usage from DLRN server in https://trunk.rdoproject.org/. This includes:

• consistent repos from http://trunk.rdoproject.org/centos7-<release>/consistent/
• current-passed-ci repos from http://trunk.rdoproject.org/centos7-<release>/current-passed-ci/
• current-tripleo repos from http://trunk.rdoproject.org/centos7-master/current-tripleo/
• By-commit repos http://trunk.rdoproject.org/centos7-<release>/AB/CD/ABCD…Z (where AB are the two first characters of the commit hash, CD the following two and ABCD…Z the entire hash). On each hash repository directory a delorean.repo file is created containing the yum configuration required to enable it easily.
• html formated build reports and build logs to help packagers to troubleshoot issues that may arise.

More info

RDO Packaging guide
Delorean: OpenStack packages from the future by Frederic Lepied
DLRN documentation
Testing Openstack is Hard by David Moreau Simard

As a new contributor to the RDO project, one of my first tasks was to understand what is actually being produced by the project. According to RDO faq, the main goal is to maintain a freely-available, community-supported distribution of OpenStack based on RPM packages able to run Red Hat Enterprise Linux, CentOS, Fedora, and their derivatives. But the RDO community is diverse and different use patterns have been found over the time with different requirements:

• Organizations running production OpenStack clouds with community support require a stable distribution.
• Organizations running production environments with community or enterprise-grade distributions require sometimes an up-to-date distribution to test recent functionalities and plan the future of their cloud deployments.
• Developers require up-to-date distribution to test their changes and run their CI jobs.
• Packagers need to have full control over the packaging process.

Providing a suitable distribution for all these scenarios is not an easy task and the RDO community ended up with two different distribution set of repos targetting different purposes:

RDO CloudSIG repos

Provide a set of stable OpenStack packages through CentOS CloudSIG repos based on CBS, CentOS Community Build System:

• The RDO CloudSIG repos are only published after GA of a upstream release (only stable branches are used, not master).
• New packages are created only when a new point release is published (release tag created) on upstream stable repositories. The RDO community is currently working to automate as much as possible the process to build these packages in rpmfactory.
• In addition to the vanilla upstream code, some patches may be applied during packaging:
  • Fixes for security issues or critical bugs not backported upstream. Note that an upstream-first policy is applied so these patches will be applied only after merged in upstream master.
  • Patches required for the packaging process.
• Packages are created through a controlled workflow. This is accomplished by koji (the technology behind CBS) which ensures reproducibility of the building process by:
  • Creating a self-contained build environment (buildroot) that provides all the dependencies by a controlled package. No network access exist int this build environment.
  • Recording both the spec file defining the package to be built and the packages included in the buildroot. This information allows to recreate exactly the same build environment to create the package at a later point in time.
• The criteria for pubishing any new release includes both thechnical (a set of CI jobs must pass as aggreed in RDO community meeting) and non-technical requirements (documentation, etc…).

This is probably what you want to use for your stable cloud with a community-supported OpenStack distribution and what you will use following the official upstream install guide or RDO docummentation to deploy and manage OpenStack.

RDO Trunk repos

Provide RPM package repositories for OpenStack projects as close as possible to the vanilla upstream code in order to test any change and catch any bug or impact on packaging as soon as possible. These packages are built when a commit is merged in upstream master or any of the supported stable branches. This means that new packages are provided even during the development phase of a new release (before GA). Once a build is done, a set of CI jobs is passed to validate installation and basic functionalities. As part of RDO Trunk we provide different repos for CentOS 7 under http://trunk.rdoproject.org/centos7-<release>/ :

• consistent: provides the latest snapshot of packages successfully built (i.e. https://trunk.rdoproject.org/centos7-master/consistent/).
• current-passed-ci: provides the latest snapshot of packages built and with CI jobs successfully passed (i.e. https://trunk.rdoproject.org/centos7-master/current-passed-ci/).
• current-tripleo: provides the latest snapshot of packages tested by specific TripleO CI jobs on master branch (https://trunk.rdoproject.org/centos7-master/current-tripleo/)

Currently (may ‘16’), <release> can be master, mitaka or liberty.

As operators, these repos may not be the best option to run a stable OpenStack cloud but it can be a good choice in some cases:

• You are interested in deploying an environment to test latest changes in OpenStack code
• You need to quickly apply a patch that has been merged upstream and can’t wait until it’s included in a point release and is properly packaged in RDO CloudSIG repos. Note that “current-passed-ci” repos are recommended in this case.

If you are a OpenStack developer, using RDO trunk may be a good option to build your environments with latest vanilla upstream code.

Finally, if you are a packager, you probably want to know how the build process went for a specific commit or even to perform an installation based on it. RDO also provides both build reports and RPM repos based on each commit for both CentOS and Fedora from http://trunk.rdoproject.org.

Fred wrote an excelent post describing how DLRN (f.k.a. delorean) builds packages for RDO trunk repos.

If you are interested in the details about how to build RDO packages, you’ll find a lot of interesting information in RDO documentation.

And remember that if you are looking for enterprise-level support, or information on partner certification, Red Hat also offers Red Hat Enterprise Linux OpenStack Platform.

As the RDO development continues to grow, the numbers of bugs also grow. To make the community robust and heavy, it is also necessary to triage and fix the existing bugs so that product will be more valuable.

We had hosted RDO Bug Triage Day on 18th and 19th May, 2016.

The main aim of the triage day was :

• Mass Closing of “End of Life” Bugs through automation with proper EOL statement. On 19th May, 2016 RDO meeting, after evaluating the Fedora EOL statement, we came up with this : “This bug is against a Version which has reached End of Life. If it’s still present in supported release (http://releases.openstack.org), please update Version and reopen.” Thanks to Rich Bowen for proposing it,
• Analysis the bug and assign proper component, target-release and ask for more information if the bug is incomplete
• Close the Bug if it is already fixed in the current release
• Provide patches to the existing Bugs.

Those 2 days was awesome for RDO.

Here is some of the stats of RDO Bug Triage Day:

• 433 Bugs affected on Bug Triage Day.
• 398 Bugs closed on Bug Triage Day out of which 365 EOL Bugs closed by automation script.
• 35 Bugs triaged on Bug Triage Day and most of the EOL triaged bugs closed due to EOL automation script
• 22 people participated in the RDO Bug Triage Day.

Thanks to Pradeep Kilambi, Peter Nordquist, Javier Pena, Ivan Chavero, Matthias Runge, Suraj Narwade, Christopher Brown, Dan Prince, Mike Burns, Dmitry Tantsur, Alfredo Moralejo, Alan Pevec, Miroslav Suchy, Masanari Matsuoka, Garth Mollett, Giulio Fidente, David Moreau Simard, Chandan Kumar and Emilien Macchi for participating and making the RDO Bug triage day successful.

The above stats is generated using bztraigeday_stats.py script.

Here’s what RDO engineers have been blogging about over the last week:

Connecting another vm to your tripleo-quickstart deployment by Lars Kellogg-Stedman

Let’s say that you have set up an environment using tripleo-quickstart and you would like to add another virtual machine to the mix that has both “external” connectivity (“external” in quotes because I am using it in the same way as the quickstart does w/r/t the undercloud) and connectivity to the overcloud nodes. How would you go about setting that up?

… read more at http://tm3.org/6t

Reproducing an Open vSwitch Bridge Configuration by Adam Young

In the previous post, I described the setup for installing FreeIPA on a VM parallel to the undercloud VM setup by Tripleo Quickstart. The network on the undercloud VM has been setup up by Ironic and Neutron to listen on a network defined for the overcloud. I want to reproduce this on a second machine that is not enrolled in the undercloud. How can I reproduce the steps?

… read more at http://tm3.org/6u

ARA: An idea to store, browse and troubleshoot Ansible Playbook runs by David Moreau Simard

Ansible can be used for a lot of things and it’s grown pretty popular for managing servers and their configuration. In the RDO and OpenStack communities, Ansible is heavily used to deploy or test OpenStack through Continuous Integration (CI). Projects like TripleO-Quickstart, WeIRDO, OpenStack-Ansible or Zuul v3 are completely driven by Ansible.

… read more at http://tm3.org/6v

I’ve been traveling a lot over the last few weeks, and have fallen behind on the blog post updates. Here’s what RDO enthusiasts have been blogging about since OpenStack Summit.

I posted a number of “What Did You Do In Mitaka” interview posts, so here those are, all together:

• Ivan Chavero
• Ihar Hrachyshka
• Chandan Kumar
• Haïkel Guémar
• Javier Peña
• Emilien Macchi
• David Moreau Simard
• Adam Young

Additionally, there were the following:

Deploying the new OpenStack EC2 API project by Tim Bell

OpenStack has supported a subset of the EC2 API since the start of the project. This was originally built in to Nova directly. At CERN, we use this for a number of use cases where the experiments are running across both the on-premise and AWS clouds and would like a consistent API. A typical example of this is the HTCondor batch system which can instantiate new workers according to demand in the queue on the target cloud.

… read more at http://tm3.org/6e

Running Keystone Unit Tests against older Versions of RDO Etc by Adam Young

Just because upstrem is no longer supporting Essix doesn’t mean that someone out there is not running it. So, if you need to back port a patch, you might find yourself in the position of having to run unit tests against an older version of Keystone (or other) that does not run cleanly against the files installed by tox.

… read more at http://tm3.org/6f

Containers and the CERN cloud by Ricardo Rocha

In recent years, different groups at CERN started looking at using containers for different purposes, covering infrastructure services but also end user applications. These efforts have been mostly done independently, resulting in a lot of repeated work especially for the parts which are CERN specific: integration with the identity service, networking and storage systems. In many cases, the projects could not complete before reaching a usable state, as some of these tasks require significant expertise and time to be done right. Alternatively, they found different solutions to the same problem which led to further complexity for the supporting infrastructure services. However, the use cases were real, and a lot of knowledge had been built on the available tools and their capabilities.

… read more at http://tm3.org/6g

Meet Red Hat OpenStack Platform 8 by Sean Cohen

Last week we marked the general availability of our Red Hat OpenStack Platform 8 release, the latest version of Red Hat’s highly scalable IaaS platform based on the OpenStack community “Liberty” release. A co-engineered solution that integrates the proven foundation of Red Hat Enterprise Linux with Red Hat’s OpenStack technology to form a production-ready cloud platform, Red Hat OpenStack Platform is becoming a gold standard for large production OpenStack deployments. Hundreds of global production deployments and even more proof-of-concepts are underway, in the information, telecommunications, financial sectors, and large enterprises in general. Red Hat OpenStack Platform also benefits from a strong ecosystem of industry leaders for transformative network functions virtualization (NFV), software-defined networking (SDN), and more.

… read more at http://tm3.org/6h

OpenStack Summit Austin: Day 1 by Gordon Tillmore

We’re live from Austin, Texas, where the 13th semi-annual OpenStack Summit is officially underway! This event has come a long way from its very first gathering six years ago, where 75 people gathered to learn about OpenStack in its infancy. That’s a sharp contrast with the 7,000+ people in attendance here, in what marks Austin’s second OpenStack Summit, returning to where it all started!

… read more at http://tm3.org/6i

OpenStack Summit Austin: Day 2 by Gordon Tillmore

Hello again from Austin, Texas where the second busy day of OpenStack Summit has come to a close. Not surprisingly, there was plenty of news, interesting sessions, great discussions on the showfloor, and more.

… read more at http://tm3.org/6j

Culture and technology can drive the future of OpenStack by E.G.Nadhan

“OpenStack in the future is whatever we expand it to”, said Red Hat Chief Technologist, Chris Wright during his keynote at the OpenStack Summit in Austin. After watching several keynotes including those from Gartner and AT&T, I attended other sessions during the course of the day culminating in a session by Lauren E Nelson, Senior Analyst at Forrester Research. Wright’s statement made me wonder about what lies in store for OpenStack and where would the OpenStack Community — the “we” that Wright referred to — take it to in the future.

… read more at http://tm3.org/6k

OpenStack Summit Austin: Day 3 by Gordon Tillmore

Hello again from Austin, Texas where the third day of OpenStack Summit has come to a close. As with the first two days of the event, there was plenty of news, interesting sessions, great discussions on the showfloor, and more. All would likely agree that the 13th OpenStack Summit has been a Texas-sized success so far!

… read more at http://tm3.org/6l

Resource management at CERN by Tim Bell

As part of the recent OpenStack summit in Austin, the Scientific Working group was established looking into how scientific organisations can best make use of OpenStack clouds.

… read more at http://tm3.org/6m

OpenStack Summit Austin: Day 4 by Gordon Tillmore

Hello again from Austin, Texas where the fourth day of the main OpenStack Summit has come to a close. While there are quite a few working sessions and contributor meet-ups on Friday, Thursday marks the last official day of the main summit event. The exhibition hall closed its doors around lunch time, and the last of the vendor sessions occurred later in the afternoon. As the day concluded, many attendees were already discussing travel plans for OpenStack Summit Barcelona in October!

… read more at http://tm3.org/6n

OpenStack Summit Newton from a Telemetry point of view by Julien Danjou

It’s again that time of the year, where we all fly out to a different country to chat about OpenStack and what we’ll do during the next 6 months. This time, it was in Austin, TX and we chatted about the new Newton release that will be out in October.

… read more at http://tm3.org/6o

Identity work for the OpenStack Newton release by Adam Young

The Newton Summit is behind us, and we have six months to prepare for the next release in both upstream OpenStack and RDO. Here is my attempt to build a prioritized list of the large tasks I want to tackle in this release.

… read more at http://tm3.org/6p

Mitaka Cinder Recap sessions by Gorka Eguileor

During Mitaka we introduced some big changes in Cinder that have a great impact for developers working on new and existing functionality. These new features include, but are not limited to, API microversions, support for Rolling Upgrades, and conditional DB update functionality to remove API races. So we decided to have Recap Sessions during the OpenStack Summit in Austin.

… read more at http://tm3.org/6q

Analysis of techniques for ensuring migration completion with KVM by Daniel Berrange

Live migration is a long standing feature in QEMU/KVM (and other competing virtualization platforms), however, by default it does not cope very well with guests whose workload are very memory write intensive. It is very easy to create a guest workload that will ensure a migration will never complete in its default configuration. For example, a guest which continually writes to each byte in a 1 GB region of RAM will never successfully migrate over a 1Gb/sec NIC. Even with a 10Gb/s NIC, a slightly larger guest can dirty memory fast enough to prevent completion without an unacceptably large downtime at switchover. Thus over the years, a number of optional features have been developed for QEMU with the aim to helping migration to complete.

… read more at http://tm3.org/6r

What did everyone do for the Mitaka release of OpenStack ? by David Moreau Dimard

Just what did everyone do for the Mitaka OpenStack release ? RDO community liaison Rich Bowen went to find out. He interviewed some developers and engineers that worked on OpenStack and RDO throughout the Mitaka cycle and asked them what they did and what they were up to for the Newton cycle.

… read more at http://tm3.org/6s

In this installment of our “What did you do in Mitaka” series, I speak to Adam Young about Keystone.

(If the above player doesn’t work for you, please download the audio HERE.)

Rich: I’m speaking with Adam Young , who’s been working on the Keystone project for the last 4 years or so.

So, Adam, what did you do in Mitaka?

Adam: This release, I finally got a couple key features into Keystone for dealing with some security issues that have bothered me for a while. The biggest one is the ability to deal with admin - the fact that if you’re admin somewhere you end up being admin everywhere.

This is bug 968696. Yes, I have the number [memorized]. I even had tshirts made up for it. We’re finally closing that bug. Although closing it properly will take several releases.

The issue with bug 968696 is that we have no way of splitting out what certain APIs are that are supposed to be admin-specific. Like, you can see that there’s a difference between adding a new hypervisor, and going into a project and creating a virtual machine. They are different levels of administration. There are certain APIs that need to have admin, that are really project-scope things. Things like adding a user to a project - role assignment - that’s admin level at the project, as opposed to admin level at the overall service.

So what we did is we put a config option in Keystone which lets you say that a certain project is the adminstrative project. Tokens that are issued for a user that are scoped to that project have an additional field on them. is_admin_project. This can be enforced in policy across OpenStack. It does mean that we need to rewrite the policy files. We knew this was going to be a long slog.

There’s a way of now being able to say on a given policy rule, this is not just that they have to have the admin role, but that they have to have the admin role on a project that’s considered the admin project.

It also allows you to say, if I have admin on the admin project, I can go into a non-admin project, and do these administrative type things. But somebody who is admin on some other common project, some other regular project, does not have the ability to do that. So the projects continue to be the main scoping for access control, but we have this level of defining things to be cloud- or service-level administration.

So that was one of the big ones. We’ll continue to drive on with using that - making that policy change throughout Nova, Glance, and the rest, because if they’re not enforcing it, then it is still a global admin. So there’s more work to be done there.

The other big feature falls into a similar type of problem-set, which is, roles are very, very coarse grained. If you look at the early days of OpenStack, there really was only one role, which was admin. And then everybody else was a member of a project.

By the time I started, which was four years ago, there already was the idea that a role assignment was for a role on a project. Role assignments were no longer global, even though they had been in the past. A lot of people still kept treating them like they were global things, but the mechanism scoped it to a project. What we wanted to be able to is say, I want to be able to have a different role for different operations - different workflows. So that I can assign small things, and I can delegate smaller pieces of what I need to do to a user.

A good example is that I might not want somebody to be able to create a virtual machine, or some offline service to be able to create a virtual machine on by behalf, but I want them to be able to reboot one of mine. They can monitor, and if it locks up they should be able to reboot that virtual machine.

So what we have is this concept of an implied role, or role inference rule.

The first one, and the example that everyone kind of gets is, if I’m an admin on a project, I should be a member on that project. So now, when I have a token and I reqest admin on there, I will see both the admin role and the member role inside that project, even if I don’t have the member role explicitly assigned to me.

Now in the API I could specify, you only need the member role to do this. So now we’ve just made it easier to roll up finer-grained permissions to a more coarse-grained role assignment.

This, it turned out, was a feature that somebody else could use to implement something they were talking about doing. Henry Nash, who works at IBM, had this proposal for domain-specific roles. He wanted to be able to have somebody who’s in some organization, specify what the name of the role was that they gave to people, and it also included a subset of what the person could do there.

I said, that really sounds like a role inference rule, but one where a) the top-level name is assigned by somebody who’s not a global cloud administrator, the domain administrator, and b) it doesn’t show up in the token. I don’t want domain-specific roles to be something that people are enforcing policy on. I want it to be [only enforce on] the global roles.

So we worked together and came up with this idea that we get implied roles in first, and then we would build the domain-specific roles on top of that.

It took a lot of collaboration - a lot of arguing and a lot of understanding … to explain to the other person what our problems were, and coming to the idea that a very elegant mechanism here could be the baseline for both features.

So we’re presenting on both of those things together, actually, on Tuesday at Summit. (OpenStack Summit, Austin - video here ) The Nash and Young reunion tour. We couldn’t get either Stills or Crosby to show up.

Beyond that, Keystone is, as somebody once described it, performance art. Getting Keystone features in requires a lot of discussion. It’s really fundamental to OpenStack that Keystone just work.

A lot of people will file bugs against Keystone, because it’s the first thing that reports a failure. Or someone reports a failure trying to talk to Keystone. So a lot of time has been spent in troubleshooting other people’s problems, connectivity problems, configuration problems, and getting it so that they understand, yes, you can’t talk to Keystone, but it’s because you have not configured your service to talk to Keystone.

And one of the big victories that we had is that a whole class of errors that we had were due to threading issues in eventlet, and eventlet is dead. Eventlet has been deprecated for a while. In TripleO we have it so that Keystone is not being deployed in eventlet any more. This means that we don’t have the threading issues there.

A big lesson that we’ve learned is that the cloud deployer does not necessarily own the user database. And for large organizations, there already is a well-established user database, and so we want to reuse … a lot of people point to stuff like Facebook, and Google, and LinkedIn, as public authentication services now. OpenID, OAuth, and all those, are protocols that make it possible to consume these big databases of identity, and then you can use your same credentials that you use to talk to Facebook, to talk to your apps.

Well, this same pattern happens a lot inside the enterprise, and in fact it’s the norm. It’s called Single Sign On, and a lot of people want it. And pushing Single Sign On technologies, and making those work within Keystone has been a long-running task.

There’s been a lot of LDAP debugging. It’s still one of the main pain points for people. LDAP is a primary tool for most large organizations for managing the users. The D in there stands for Directory and it’s user directory for large organizations. A lot of people have to make it work with Active Directory, Microsoft being so dominant in the enterprise, and Active Directory being their solution for that has made it a pain point. So one of the big things we’ve been trying to do is make that integrate in better with existing deployments.

Over multiple releases we had this idea that first you had one monolithic identity back-end. And then we split that out into two pieces, one which was the identity, the users and the groups, and the other which was the assignment - what roles you had on each project.

Now that we can vary those two things differently, what we started doing is saying, how about having multiple things on that identity side? Federation kind of works in there but before that we actually said you could have SQL back your user database, and that would be your basic install. All the service users for Keystone, and Nova and so on would be put in there, and then let’s make a separate domain and put LDAP in there. So the big push has been to do that.

Well it turns out that in order to do that we need everybody to use the Keystone V3 API, because they need domain support. Specifically internally, the different services need to do this.

A lot of people have helped make this happen. This has been a multi-project effort because not only do we need to solve it within all the Keystone components, but then there are places where one service calls into another and needs to authenticate with Keystone and those calls there are calling through client network access.

So we’re finally at a place where V3 Keystone API everywhere is a reality, or can be a reality, and so getting the pain of LDAP into its own domain has been an ongoing theme here. I really feel it’s going to make things work the way people want them to work.

One of the nice benefits of this, especially for new deployments, is they no longer have to put service users into LDAP. This was a dealbreaker for a lot of people. LDAP is a read-only resource. If, say, the OpenStack user does not own the user database, that includes service users. So being able to have service users in a small database inside Keystone, and have consume the rest of the identity from LDAP has always been the dream.

Keystone moves very slowly. It’s a very cautious project, and it has to be.

Rich: Because it has to just work.

Adam: It has to just work, and it’s the target for security. If you can hack Keystone, then you have access to the entire cloud. And if you’re using LDAP, you have the potential to provide access to things beyond the cloud.

R: Tell me about some things that are coming in upcoming releases.

A: One that I’m looking forward to in Newton is the ability to unify all the delegation mechanisms. What we’re doing with trusts, which is something I built specifically for Heat a couple of releases ago, and have taken on this way of a user being able to delegate another user a subset of their permissions, and to be able to delegate the ability to delegate. You don’t have to be an admin in order to create some sort of delegation, but you can be an everyday user and hand off, to maybe a scheduled task, to do something on your behalf, at midnight when you’re on vacation.

The other one that I’m really looking forward to trying to drive on is the adoption of Fernet. We were hoping to get that into the Mitaka release. Fernet is in there, but trying to make it the default.

Fernet is an ephemeral token. The way that tokens work with Keystone now … it’s a UUID. It’s a claim check that you can hand to somebody and they can go back to Keystone, assuming they have authority to do this, and say, what data is associated with this token?

Well, Keystone has to record that.

If you have two different Keystone servers, and they’re both issuing out tokens, they have to talk to the same database. Because I might get one issued from one, and go to a different one to validate it.

This is a scalability issue.

PKI tokens, which I wrote many releases ago, were a first attempt to deal with this, because you could have different signers. But PKI tokens had a lot of problems. Dolph and Lance, who are Rackspacers, who have been long-term contributors, came up with this implementation called Fernet tokens. The idea is that you use symmetric cryptography. You take the absolute minimal amount of data you need to reconstitute the token and you sign it. Now, the Keystone server is the only thing that has this key. So that means it’s the only thing that can decrypt it. So you still need to pass the token back to Keystone to validate it, but Keystone no longer has to go to a database to look this up. It can now take the Fernet token itself, and say here’s what it means. And if you have two different Keystone servers, and they share keys, you can get one issued from one, and have the other one validate it for you.

R: Well, thank you again for taking the time to do this.

A: My pleasure.

Continuing our series “What Did You Do In Mitaka?”, I spoke with David Moreau Simard.

(If the above player doesn’t work for you, please download the recording HERE.)

Rich: I’m speaking with David Moreau Simard, who is one of the engineers who works on RDO at Red Hat. Thanks for taking time to speak with me.

DMSimard: It’s a pleasure

R: What did you work on in Mitaka?

D: I’m part of the RDO engineering team at Red Hat. My role is mostly around continuous integration and making sure that RDO works. This means that some days I could be implementing new ways of testing RDO or some other days fixing bugs identified by our CI.

As far as Mitaka is concerned, we got Packstack to a point where it’s able to install and run Tempest against itself. This is awesome because it’s a great way to make sure everything works properly, sort of a sanity check.

This improvement allowed us to make Packstack gate against itself upstream to prevent regressions from merging by testing each commit. This was part of a plan to make Packstack part of our testing pipeline.

In Mitaka, we started using a new project called WeIRDO. It’s meant to use upstream gate integration tests inside our own RDO testing pipeline. We really improved our testing coverage in Mitaka with TripleO quickstart and WeIRDO. With WeIRDO, we’re able to easily test RDO trunk packages against Packstack and Puppet-OpenStack and the jobs that they provide.

We built a great relationship with projects that consume RDO packages throughout development cycles. We mutually benefit from essentially testing each other. So it makes a lot of sense to stay in touch with these guys.

Overall, I can really tell how the testing coverage for RDO has improved tremendously, even since liberty. Mitaka is really the best and most tested release of RDO yet. I want to extend my thanks to everyone that was involved in making it happen.

R: So how about in Newton? What do you think is coming in Newton for you?

D: So, Mitaka was already an awesome cycle from the perspective of testing coverage that we had for RDO. We want to keep working on that foundation to increase the coverage further. I also want to spend some time on improving monitoring and automation to make troubleshooting problems easier.

Trunk breaks all the time, and so it takes a lot of time to troublshoot that, and we want to make that easier. But also have visibility into the problems that could be coming.

Earlier I talked about WeIRDO and how we use upstream gate jobs in our testing pipeline. In Newton, we want to add Kolla back into our testing pipeline through WeIRDO. I had some issues getting Kolla to work reliably in Mitaka but they’ve fixed some problems upstream since then so we’ll take another look at it.

Just recently, there’s also the Chef OpenStack community – they’ve been working on integration testing their cookbooks in the gate. I’m definitely interested in seeing what they have, and possibly leveraging their work once they’re ready.

The other large chunk of work will probably come from our effort in streamlining and centralizing the different components of RDO. Right now, we’re a bit all over the place… But we’ve already started working towards moving to a software factory instance. Software factory is a project to provide upstream’s testing ecosystem with Git, Zuul, Nodepool, Gerrit and Jenkins.. but in a box, an appliance.

This will definitely make things easier for everyone since everything from RDO will be expected to be in this one place.

R: Thank you very much for your time.

D: I appreciate yours, and it was great working on this great release.

David is dmsimard on Twitter, github, and on IRC.

Continuing my series of interviews with RDO engineers, here’s my chat with Emilien Macchi, who I talked with at OpenStack Summit last week in Austin.

If you’d like to talk about what you did in Mitaka, please contact me at rbowen@redhat.com.

(If the above player doesn’t work for you, please download the audio HERE.

Rich: I’m speaking with Emilien Macchi, who is one of the people that worked on the Mitaka release of OpenStack. Thank you for taking time to speak with me.

Emilien: Thank you for welcoming me.

R: What did you do in Mitaka?

E: In Mitaka, most of my tasks were related to packaging testing. I’m currently leading the Puppet OpenStack group, which is a team where we develop some Puppet modules that deploy OpenStack in production. Red Hat is using the Puppet modules in TripleO, so we can automate the deployment of OpenStack services. Recently we switched the gates of OpenStack Infra to use RDO packaging. We had a lot of issued before because we didn’t have much CI. But since Mitaka, the RDO and the Puppet teams worked a lot together to make sure that we can provide some testing together. We can fix things very early when it’s broken upstream. Like when we have a new dependency or when we have a change in the requirements. That was this kind of test where you provide very early feedback to the RDO people that are doing the packaging. It was very interesting to work with RDO.

We provided early feedback on the builds, on the failures. We also contributed to … I say “we”, I don’t like to say “I” … we, my group, my Puppet group, we contributed to the new packages that we have in Mitaka. We updated the spec in the package so we could have more features, and more services in the RDO projects.

Testing! Testing every day. Testing, testing, testing. Participating in the events that the RDO community is having, like the weekly meetings, the testing days. It was very interesting for us to have this relationship with the RDO team.

So I’m looking forward to Newton.

R: What do you anticipate you’ll be working on in Newton?

E: Well, that’s an interesting question. We are talking about this week. [At OpenStack Summit, Austin.]

The challenge for us … the biggest challenge, I guess, is to follow the release velocity of upstream projects. We have a bunch of projects in OpenStack, and it’s very hard to catch up with all of the projects at the same time. In the meantime we have to because those projects, they are in the RDO project and also in the products, so that’s something we need to … we need to scale the way that we producing all this work.

Something we are very focused on right now is to use the Software Factory project, which is how OpenStack Infra is operating to build OpenStack software. This is a project - an open source project - which is now used by RDO. We can have this same way as upstream, but for downstream - we can have the same way to produce the RDO artifacts, like the packages and the Puppet modules, and so on. That’s the next big challenge, is to be fully integrated with Software Factory, to implement the missing features we will need to make the testing better.

Yeah, that’s the biggest challenge: Improve the testing, and stopping doing manual things, automate all the things, that is the next challenge.

R: Thank you very much. Thanks for your time.

E: Thank you for your time.

After releasing RDO Mitaka, we migrated our current packaging workflow from gerrithub to a brand-new platform hosted at https://review.rdoproject.org right before the Newton Summit. During the last cycle, we’ve worked with Software Factory folks to build a new packaging workflow based on the very same foundations as upstream OpenStack (using gerrit, zuul, nodepool). We’re now relying on RPMFactory which is a specialization of Software Factory fitted for RPM packaging needs.

The migration was planned in order to satisfy some criteria:

• providing a consistent user experience to RDO contributors
• simplify and streamline our workflow
• consolidate resources

Workflow changes

So now, packages are maintained through https://review.rdoproject.org and packages dist-git are replicated to the rdo-packages github organization. The rdopkg utility has been updated to support the new workflow with new commands: clone, review-patch and review-spec.

Anyone can submit changes for the review, which can be approved by the following groups:

• rdo-provenpackager much like Fedora provenpackagers can +2 and +W changes for all projects.
• PROJECT-core: package maintainers listed in rdoinfo can +2 and +W changes for their projects.

Howto

We’re working to refresh documentation, but here’s a short howto to hack on packaging:

1. dnf copr enable jruzicka/rdopkg && dnf install -y rdopkg
2. rdopkg clone [-u ] This will create the following remotes
   • origin: dist-git
   • patches: upstream sources at current package version
   • upstream: upstream sources
   • review-origin: used to interact with RPMFactory
   • review-patches: used to interact with RPMFactory
3. modify packaging
4. rdopkg review-spec
5. review then merge

For downstream patches, the workflow is a bit different as we don’t merge patches but keep reviews open in order to keep the whole patch chain history. rdopkg has been updated to behave properly in that aspect.

We still have improvements coming but no major changes to this workflow, if you have any questions, feel free to ping us on irc (#rdo @freenode) or the mailing list.

Regards, The RDO Eng. team