Before releasing Mitaka, we agreed on a tecnical definition of done1. This can always evolve to add more coverage, but this is what we currently test when deciding whether a release is ready from a technical perspective:

• Packstack all-in-one deployments testing the 3 upstream scenarios
• TripleO single contoller and single compute deployment validated with Tempest smoke tests
• TripleO three controller and single compute deployment with pacemaker validated using a Heat template based ping test

These same tests are used to validate our trunk repos2.

OpenStack engineers met in Austin last week to design the Newton cycle. Here is a report for the Security Project and other associated efforts.

Requirements for the vulnerability:managed governance tag

During Mitaka, the vulnerability management team (“VMT”) introduced a new vulnerability:managed tag to help identify supported projects. In essence, this tag indicates that a project’s security bug reports are triaged by the VMT taxonomy. For class A reports, the VMT issues an OpenStack Security Advisory (“OSSA”).

The VMT documented a set of requirements to get the tag:

• Deliverable repositories,
• Dedicated point of contact, usualy a subset of the core team called coresec,
• PTL agrees to act as a VMT liaison to escalate severe issues,
• Bug tracker reconfiguration, and
• Third-party review/audit/threat analysis.

The last point (item 5) intends to ensure that the VMT doesn’t become a bottleneck for projects with poor security practices or that have unknown fundamental flaws. By design, the VMT is a small team, composed of 3 volunteers. Thus, new projects must demonstrate that they won’t cause an unmanageable number of security bug reports.

In the next section, I will set-out how we can efficiently increase the number of vulnerability:managed projects as designed during the Newton summit etherpad.

What VMT members need to know

Since day 1, doing VMT work for OpenStack has been a challenging task due to:

• A wide attack surface,
• Source code and api version changes,
• Deployment modes with distinct threats, and
• Coordination between different communities.

Here is the list of questions that need to be answered when managing a new project:

• What are the input data formats and how are they processed ?
• Which actions are restricted to operators ?
• What has been considered a vulnerability in previous bug reports ?
• What can a malicious user do depends on:
  • Available services, including the non-obvious ones, such as nova-novncproxy,
  • API endpoint paths, including old versions, such as glance v1, and
  • Every other network-facing service to which a user can connect.

A document answering the above questions will likely be required for any vulnerability:managed tag application. Moreover, already supported projects will also need to validate the above requirements in order to keep the tag.

This document needs to be maintained within the OpenStack community using open code review. The security-guide or project documentation are appropriate locations to publish it.

Here is the proposed threat analysis process and Anchor’s threat analysis.

Stable Release

Major changes to the OpenStack release process were introduced last cycles:

• Integrated releases, such as 2015.1.2 are now gone,
• Projects may use different release models,
• Unified versions numbers semver, and
• Automated publication to website.

The “stable summit” set new goals for the Newton cycle:

• Increase stable release support to 18 months with an option for 6 additional months,
• Backwards compatibility for libraries, and
• Co-installability requirements.

End of life policy is the most important topic for the VMT since it increases the number of supported branchs. Infra and QA are also directly affected since they have to maintain the CI capacity for supported stable releases. While in theory it’s possible to extend support, here is a list of practical blockers to be addressed:

• Stable branch gates break too often, see openstack-health,
• Backport to old releases is time consuming when the affected code went through several refactors, and
• Long Term Stable (LTS) versions aren’t an option because upgrade process must deploy each versions.

In summary, Kilo release end of life won’t be extended and the branch will likely be closed soon, EOL: 2016-05-02. Further, external dependencies requirements are not supported by the VMT, but it is worth noting that a new Requirement Team will be created to manage this increasingly difficult situation.

More details in those two etherpads: eol-policy and stable summit

Issue tracker

Last but not least, there was a design session to discuss issue tracker for OpenStack projects. This is a critical topic for vulnerability management since most VMT work actually happens on the issue tracker.

OpenStack projects currently use launchpad.net to track issues and write blueprints for feature tracking. The workflow being sub-optimal, there is an on-going effort to replace launchpad by something better. Migration issues aside, the real concern is that we may end up with another sub-optimal solution which doesn’t justify the migration cost.

On the other hand, the OpenStack community initiated storyboard which is designed to address the exceptional needs of OpenStack development. Unfortunately, the project stalled due to the lack of reviewers.

The OpenStack community has three main options regarding issue tracking:

• Develop its own issue tracker (e.g. storyboard),
• Operate a ready-to-use solution (e.g. maniphest), or
• Use an external service (e.g. launchpad).

This topic will be discussed in the following weeks. Hopefully it will lead to a long term decision from the TC with Infra approval.

Conclusion

Once again, we have made great progress and I’m looking forward to further developments. Thank you all for the great OpenStack Newton Design Summit!

Tristan Cacqueray

OpenStack Vulnerability Management Team

On Tuesday afternoon about 60 people gathered for the RDO BoF (Birds of a Feather) session at OpenStack Summit in Austin.

By the way, the term Birds of a Feather comes from the saying “Birds of a feather flock together”, which refers to like-minded people gathering.

The topics discussed can be found in the agenda.

The network was friendly, and we managed to capture the whole session in a Google Hangout. The video of the session is HERE.

And there are several pictures from the event on the RDO Google+ page.

Thank you for all who attended. You’ll see follow-up discussion on a number of the topics discussed on rdo-list over the coming weeks.

We’re continuing our series of “What did you do in Mitaka?” interviews. This one is with Javier Peña.

(If the above player doesn’t work for you, you can download the file HERE.

Rich: Today I’m speaking with Javier Peña, who is another member of the Red Hat OpenStack engineering team. Thanks for making time to speak with me.

Javier: Thank you very much for inviting me, Rich.

R: What did you work on in the Mitaka cycle?

J: I’ve been focusing on three main topics. First one was keeping the DLRN infrastructure up and running. As you know, this is the service we use to create RPM packages for our RDO distribution, straight from the upstream master commits.

It’s been evolving quite a lot during this cycle. We ended up with so much success that we’re having infrastructure issues. One of the topics for the next cycle will be to improve the infrastructure, but that’s something we’ll talk about later.

These packages have now been consumed by the TripleO, Kolla, and Puppet OpenStack CI, so we’re quite well tested. Not only are we testing them directly from the RDO trunk repositories, but we have external trials as well.

On top of that I have also been working on packaging, just like some other colleagues on the team who’ve you’ve already had a chance to talk to - Haikel, Chandan - I have been contributing packages both to upstream Fedora, and also RDO directly.

And finally, I’m also one of the core maintainers of Packstack. In this cycle we’ve been adding support for some services such as AODH and Gnocchi. Also we switched Mysql support from the Python side. We switched libraries, and we had to do some work with the upstream Puppet community to make sure that PyMysql, which is now the default Python library used upstream, is also used inside the Puppet core and we can use it in Packstack.

R: You mentioned briefly infrastructure changes that you’ll be making in the upcoming cycle. Can you tell us more about what you have planned for Newton?

J: Right now the DLRN infrastructure is building 6 different lines of repositories. We have CentOS master, which is now Newton. Mitaka, Liberty, and Kilo. And we have two branches for Fedora as well. So this is quite a heavy load for VMs that we’re running right now. We were having some issues in the cloud we were running that instance. So what we’re doing now is we are migrating to the CentOS CI infrastructure. We are having a much bigger machine in there. And also what we are going to do is we will be publishing the resulting repositories using the CentOS CDN, which is way more reliable than what we could build with individual VMs.

R: Thank you again for your time. And we look forward to seeing what comes in Newton.

J: Thank you very much, Rich.

In this installment of the “What Did You Do in Mitaka” series, I’m speaking with Haïkel Guémar

(If the above player doesn’t work for you, you can download the audio HERE.)

Rich: Thanks for making time to speak with me.

Haïkel: Thank you.

R: So, tell me, what did you do in Mitaka?

H: I work on the RDO engineering team - the team that is responsible for the stewardship of RDO. For this cycle, I’ve been focusing on our new packaging work flow.

We were using, for a long time, a piece of infrastructure taken from Fedora, CentOS, and GitHub. This didn’t work very well, and was not providing a consistent experience for the contributors. So we’ve been working with another team at Red Hat to provide a more integrated platform, and one that mirrors the one that we have upstream, based on the same tools - meaning Git, Gerrit, Jenkins, Nodepool, Zuul - that is called Software Factory. So we’ve been working with the Software Factory team to provide a specialization of that platform called RPMFactory.

RPMFactory is a platform specialized for producing RPMs in a continuous delivery fashion. It has all the advantages of the old tooling we have been using, but with more consistency. You don’t have to look in different places to find the source code, the packaging, and stuff like that. Everything is under a single portal.

That’s what I’ve been focusing on during this cycle, on top of my usual duties, which is producing packages and fixing it.

R: And looking forward to the Newton release, what do you think you’re going to be working on in that cycle?

H: While we’ve been working in the new workflow, we’ve been setting a new goal, that is to decrease the gap between upstream releases and RDO releases down to 2 weeks. Well, we did it on the very same day for Mitaka! So my goal would be for Newton to do as good as this time, or even better. Why not under 2 hours? Not putting on ourselves more pressure, but to try to release almost at the same time as upstream GA, RDO. And also with the same quality standards.

One of the few things that I was not happy with during the Mitaka cycle was mostly the fact that we didn’t manage to release some packages in time, so I’d like to do better. Soon enough I will be asking for people to fill in a wish list on packaging, so that we are able to work on that earlier. And so we could release them on time with GA.

R: Thanks again for taking the time to do this.

H: Thank you Rich, for the series.

Next in our series of “What Did You Do In Mitaka” articles, I spoke with Chandan Kumar, who works on packaging for RDO, and also is active on the mailing list and on IRC.

See also, Ivan Chavero and Ihar Hrachyshka’s interviews.

(If the above player doesn’t work for you, you can download the podcast recording HERE.)

Rich: RDO Mitaka was released a little over a week ago. I’m speaking with Chandan Kumar, who is one of the people who is very active in the RDO community. Thanks for taking time to speak with me.

Could you tell me what you did during the Mitaka cycle?

Chandan: Thank you Rich.

Well, this release, I was packaging and reviewing RDO packages, mostly. In the start of the Mitaka release I was tracking upstream global requirements, and found some of the packages missing from the RDO packages. So I maintained and packaged them.

In the list of packages are: python-wsgi_intercept, python-reno, python-tosca-parser, python-weakrefmethod, python-XStatic-roboto-fontface and many more in which Python reno is most popular because it was used for adding and deleting release notes to upstream OpenStack projects.

In the midcycle, I got a chance to work with Matthew from CERN. And at the same time I had just completed packaging Python Magum client, and helped in packaging Magnum for RDO

I also worked with Javier, Alan, and Haikel, in porting spec files of Oslo libraries and its dependencies to Python 2 and Python 3 packages. At that time there is a need of RDO specific spec templates for client OpenStack test and Oslo libraries. That we have added.

Let’s now come to the end of the Mitaka release. We have created Python OpenStack services tests sub -package for all OpenStack packages present in RDO. That was consumed by Puppet OpenStack CI, and that was a great achievement for me.

And lastly, I have done code contributions and documentation changes for DLRN.

That was an exciting release in RDO land.

R: Do you expect to continue participating at the same level in the Newton cycle?

C: Yes. Since the Newton cycle has just started, Magnum trunk DLRN builds got failed because of missing packages Python k8sclient. That I have packaged, and imported, with the help of Alan, do DLRN. Now it’s working fine.

I will continue to do packaging and code contributions to rdopkg and DLRN. Apart from that, I will continue to contribute to packstack OpenStack module, and try to add new OpenStack services to RDO ecosystem.

R: Thank you very much for taking time to speak with me.

C: Thanks, Rich.

This has been a big few weeks for RDO, and for OpenStack in general, with the Mitaka release coming out, and OpenStack Summit just around the corner.

Here’s some of what RDO enthusiasts have been blogging about in the last few days.

RDO Mitaka released by Rich Bowen

The RDO community is pleased to announce the general availability of the RDO build for OpenStack Mitaka for RPM-based distributions - CentOS Linux 7 and Red Hat Enterprise Linux. RDO is suitable for building private, public, and hybrid clouds and Mitaka is the 13th release from the OpenStack project, which is the work of more than 2500 contributors from around the world.

… read more at http://tm3.org/65

What, Why, and How is the CentOS Cloud SIG? by Rich Bowen

2 years ago we started the CentOS Cloud SIG. Progress has been slow but steady. Slow, I think, because we’ve not done a wonderful job of explaining what it’s for, and why it matters. But it has been a very important aspect of moving the RDO project forward, so it deserves better explanation.

… read more at http://tm3.org/66

Naming is hard or Who Let The Vowels Out by Alan Pevec

When Derek Higgins started the tool to build OpenStack packages based on latest upstream aka “trunk” changes, the name Delorean seemed like a good choice: the tool is keeping older builds so you could “go back in time” like with a time machine, it’s geeky enough and there’s Irish connection.

… read more at http://tm3.org/67

Integrating OpenStack into the Enterprise by Michael Solberg

Adoption of OpenStack in the enterprise has been progressing steadily over the last two years. As a Forrester Report* on enterprise adoption from September noted, “OpenStack demonstrates the completeness, robustness, and capability upon which a broader range of adopters can depend.” OpenStack deployments have proven to be complex in larger IT organizations though, but not because of the reasons that you might anticipate. Much has been made about the complexity of installing the software, but we’ve found that the lion’s share of effort in these implementation comes around the practice of integrating IaaS into the fabric of enterprise IT and evolving existing processes to meet the expectations of the user community.

… read more at http://tm3.org/68

Getting Started with Puppet for Keystone by Adam Young

Tripleo uses Puppet to manage the resources in a deployment. Puppet has a command line tool to look at resources.

… read more at http://tm3.org/69

What did you do in Mitaka: Ihar Hrachyshka talks about Neutron by Rich Bowen

In this installment of our “What did you do in Mitaka?” series, Ihar Hrachyshka talks about his work on Neutron both in Mitaka, and what’s planned for Newton.

… read (and listen) more at http://tm3.org/6a

What did you do in Mitaka? : Ivan Chavero by Rich Bowen

The OpenStack cloud software project recently released the Mitaka release. RDO is a community distribution of OpenStack. I’ve invited several of the engineers that work on RDO to talk about what they did in the Mitaka cycle.

… read (and listen) more at http://tm3.org/6b

Learning something new about Oslo Versioned Objects byGorka Eguileor

If you work on an OpenStack project that is in the process of adopting Versioned Objects or has recently adopted them, like we have in Cinder, you’ve probably done some code reviews, and even implemented a couple of Versioned Objects yourself, and are probably thinking that you know your way around them pretty well, but maybe there are some gaps in that knowledge that you are not aware of, specially if you are taking Nova’s or Cinder’s code as reference, at least I know I had some gaps. If you aren’t a seasoned Versioned Object user like the Nova people are, I recommend you keep reading.

… read more at http://tm3.org/6c

TripleO with already deployed servers by James Slagle

Recently I’ve been prototyping how to use TripleO with already deployed and provisioned servers. In such a scenario, Nova and Ironic would not be used to do the initial operating system provisioning of the Overcloud nodes. Instead, the nodes would already be powered on, running an OS, and ready to start to configure OpenStack.

… read more at http://tm3.org/6d

2 years ago we started the CentOS Cloud SIG. Progress has been slow but steady. Slow, I think, because we’ve not done a wonderful job of explaining what it’s for, and why it matters. But it has been a very important aspect of moving the RDO project forward, so it deserves better explanation.

The Cloud SIG produces packages which allow you, the CentOS user, to deploy Cloud infrastructure on top of CentOS with complete confidence that it’s going to work.

So, it’s more than just packages - the RDO project was already producing packages. It’s the entire ecosystem, testing OpenStack (and other cloud platforms) on CentOS, ensuring that OpenStack will work on CentOS, and that CentOS will work on OpenStack. To this end, there’s CI on the CentOS infrastructure, which is also used by numerous other projects, so we know that what we’re doing doesn’t break what they’re doing.

It’s also a community of cloud operators, running on CentOS, providing support to other users, and feedback and bug reports to the developers working upstream. This feedback loop further ensures the reliability of the platform, not only for CentOS users, but for the entire OpenStack community.

We meet every Thursday at 15:00 UTC on the #centos-devel channel on Freenode IRC, if you want to drop by and have your say. If you’re using CentOS, and you’re deploying any cloud infrastructure, we want to hear from you.

In this installment of our “What did you do in Mitaka?” series, Ihar Hrachyshka talks about his work on Neutron both in Mitaka, and what’s planned for Newton.

You can listen below. If the player doesn’t work for you, you can listen HERE.

Rich: Hello, I’m speaking with Ihar Hrachyshka about the work that he did in the RDO community on the Mitaka release of OpenStack. Thank you for taking time to speak with me.

Ihar: Thank you for having me

R: Could you mention the things that you’ve worked on in Mitaka, both upstream, and also specifically for RDO.

I: One thing that I was focusing on during Mitaka in upstream is - we are looking at how we can enhance the upgrade story for the project I work on, which is Neutron.

(See Neutron Mitaka release notes )

There was a lot of good work done in this regard on other projects, in previous cycles, like in Nova. And Neutron was kind of lagging on that. During Mitaka, we formed a sub-team that looks specifically into the upgrade story, and we’ve identified several pain points that we were addressing during this cycle, and plan to continue the work in Neutron, and forward.

One thing that we’ve identified is even though so-called rolling upgrades were kind of supported in the sense that there was some code to handle that scenario, for upgrades. But it was never actually validated in our CI gate, so no-one could be sure that it actually worked. So that was one thing that we tackled. We added proper jobs to validate these scenarios, and make sure that they work.

Another thing for upgrades is that we are looking at reducing and maybe even eliminating any downtime for API endpoints for Neutron, especially in the case when you have highly-available controllers. That involves a lot of work in the code base, a lot of change there to refactor the code. We’ve already started the work during Mitaka, and we’re going to proceed with that in Newton. We hope that we complete this work in Newton, and then for all upgrades starting from Newton to the next release, operators will be able to retain their Neutron API available while controllers are upgraded one by one. That’s one thing that I was working on.

There are other things. There was a lot of work done in Neutron in the scope of Quality of Service (QOS). The initial support for QOS in Neutron was merged in Liberty, but it’s still quite limited, so we expand on what we had done in that release. Specifically, the initial support was just for OpenVSwitch, but in Mitaka we added support for Linuxbridge ml2 driver. We also added role-based access control for QOS policies. We added support for so-called DSCP tags, which is a feature to prioritize traffic based on which port it came from.

Another interesting thing that we finally tackled in Mitaka is MTU support. This is one of the huge pain points that were identified by operators in the past, that Neutron, while being the networking solution for OpenStack, cannot actually properly handle MTUs - which stands for Maximum Transfer Unit. Neutron could not actually handle, neither standard MTUs for ethernet, nor so-called jumbo frames, which is really bad. It means that instances did not have access to the full capabilities of the underlying physical infrastructure. That should hopefully go away in Mitaka, because there were several changes there, both on the Neutron side, and on Nova, which now should make Neutron work out of the box, both for non-standard and standard MTU sizes.

Finally, one thing to note, apart from pure upstream work in the Master branch, is that in Neutron, in this Mitaka release, we adopted a new approach to handle backports for stable branches. Now instead of waiting for something to break in a user installation, and then reacting to their bug reports, we are proactively backporting all of the bug fixes from the latest master branch into stable branches, and release new stable releases, often. We hope that it will reduce the number of painful problems, in actual installations that rely on stable branches, very significantly.

R: This sounds like an enormous amount of work. How many people are you working with across how many organizations, would you estimate?

I: In OpenStack it’s always that you work with other people. I would say … I don’t know … there are I think 5, 6, major contributors to Neutron. Surely, nothing comes in OpenStack if you’re just single. So that’s collaboration.

R: Thank you very much for your time, and we look forward to seeing what comes in Newton.

I: Thank you.

This is the first in what will hopefully be a long series of interviews with OpenStack engineers about what they worked on in the Mitaka cycle.

The OpenStack cloud software project recently released the Mitaka release. RDO is a community distribution of OpenStack. I’ve invited several of the engineers that work on RDO to talk about what they did in the Mitaka cycle.

If the above player doesn’t appear, or doesn’t work for you, you can listen here.

Rich: Mitaka is ready now, and before we move on to Newton, we should celebrate what we’ve done in Mitaka. I’m speaking with Ivan Chavero. I understand that you worked primarily on packstack. I’ve had to correct a number of people over the last few months who think that packstack is abandoned, and we’re not doing it any more. While, at the same time, the people at CERN use packstack for a lot of things. and I know that they’re an unusual case, but there are other people that are using it.

Tell us what you did in the Mitaka cycle.

Ivan: Well, we fixed a lot of bugs. We had a lot of problems because there was some migration from Keystone version 2 to Keystone version 3. We added some other components like, Ceilometer was split into aodh and Gnocci, and we added that stuff.

I think it’s pretty stable right now.

We added more features to Neutron. Everything we have to … the new stuff, we have to test. We add it to Packstack.

I wanted to add Magnum, but it wasn’t a priority. And there’s other stuff like Ceph that we wanted to add, but it was just like, let’s fix bugs, let’s fix the stuff we need, and keep Packstack stable.

We did a lot of bug fixes for Packstack being able to work properly with the CI infrastructure. Pretty much fixing a lot of bugs that we found on the road.

R: Some of those other things that you mentioned - Magnum and Ceph and so on, are those going to be goals for the Newton cycle, then?

I: Well, I want them to be goals, but I have to discuss it with the community.

R: Who else in the community is working on Packstack?

I: There’s a lot of people. In this cycle we had a lot of contributions from outside the RDO community, which was very positive. Javier Peña is working on that. David Simard is working on that from the CI part, a lot of stuff. Alan Pevec is our guiding light. And Martin Mágr, who is the current PTL - he’s stepping down but right now he’s the PTL. A lot of people that are always asking stuff about Packstack and doing reviews, and actually submitting patches.

You know, I think that people think that Packstack is dead because it’s not recommended for production use. Only if you really know what you’re doing, you should use it in production, but it’s not recommended. TripleO is the recommended stuff for production, and Packstack is for doing the experiments, and proof-of-concept stuff.

But I use it on my home OpenStack deployment, and I do a lot of experiments. It’s pretty good for experimenting, because it’s pretty lightweight, and it’s very easy to modify.

R: Thank you very much for your time, and I look forward to what’s coming in Newton!

I: Thank you!