WireGuard + UniFi

I have been looking around for a good VPN solution to use on the road recently. I have a few services running at home, that I really don't want on the internet (OctoPrint etc), but I want to use remotely.

This is as much documentation for future me as it is for anyone who stumbles across this page :)

I had previously set up a L2TP Remote user VPN in the UniFi controller, but it had a few issues.

  • Roaming problems on mobile
  • Battery usage on mobile
  • Slow Speeds

I had heard of Wireguard a while ago (I think they had a stall near the OpenStack stall in FOSDEM last year), but I had completely forgotten about them. It turns out some kind soul has created a deb package to install WireGuard on Vyatta (which is what the USG is based on).


  • Pick up the correct .deb from here
    • curl -sL https://github.com/Lochnair/vyatta-wireguard/releases/download/<version>/wireguard-<board>-<version>.deb -o wireguard-<board>-<version>.deb worked for me
    • In my case, version was 0.0.20190123-1 and board was ugw3
  • sudo dpkg -i wireguard-<board>-<version>-1.deb to install the package
  • sudo -i to make everything easier
  • umask 077 && mkdir wireguard && cd wireguard for the server keys
  • wg genkey | tee wg_private.key | wg pubkey > wg_public.key to create server keys
  • wg genkey | tee client1_private.key | wg pubkey > client1_public.key to create the first client keys. You will need one of these keys for each client connecting to the VPN
  • Then we move over to the UniFi controller to create the config for the VPN


UniFi gateways are pretty similar to EdgeRouter products from Ubiquiti, with a crucial difference. Any config changes done from the CLI are wiped out on reboots, or any config changes from the controller. the UniFi Controller is nice, but does not support the full range of EdgeOS features that we can use.

Thankfully there is a solution - config.gateway.json. This file is layered over the base config that gets generated by UniFi, and allows much more control of a USG.

I created this file in my UniFi controller (for me, on Ubuntu the right location is /usr/lib/unifi/data/sites/<site-id>/config.gateway.json).

    "firewall": {
        "group": {
            "network-group": {
                "remote_user_vpn_network": {
                    "description": "Remote User VPN subnets",
                    "network": [
    "interfaces": {
        "wireguard": {
            "wg0": {
                "description": "VPN for remote clients",
                "address": [
                "firewall": {
                    "in": {
                        "name": "LAN_IN"
                    "local": {
                        "name": "LAN_LOCAL"
                    "out": {
                        "name": "LAN_OUT"
                "listen-port": "443",
                "mtu": "1352",
                "peer": [
                        "<content of client1_public.key>": {
                            "persistent-keepalive": 60
                "private-key": "/config/auth/wireguard/wg_private.key",
                "route-allowed-ips": "true"

TC Nomination - Rocky Cycle


I am submitting my candidacy for the OpenStack Technical Committee.

I have been contributing to OpenStack since the Havana cycle [1] [2], mainly in Designate. I have also been involved with the TC, and its meetings since Designate applied for incubation all the way back in Atlanta (the first time we were there).

Over the last 6 months I have become more involved in the TC, and have been an active contributor to TC discussions (both on IRC and in person) and governance [3].

I have been PTL for Designate for Mitaka, Newton, Ocata, Queens and Rocky cycles, and a core for a longer period. I believe my experience working in a younger, smaller project within OpenStack is a benefit. Along with the experience of working on software as an end user of OpenStack I can help us ensure the Technical Committee is mindful of the unique challenges these projects and users can face.

Read more…

Dublin PTG Summary


Not getting on a plane was a nice change for an OpenStack event :) - especially as it looks like I would not have made it home for a few days.

Cross Project Days (Monday / Tuesday)

These days are where I think the major value of the PTG is. The cross project days feel like the Summit of old, with more time for topics, and less running around a conference centre trying to cram 2 weeks worth of talks / developer sessions, and other meetings into a few days.

Unified Limits / olso.limits / keystone stored limit data

First up for me was the keystone based limits API for unifying quota data in keystone. It was decided to create oslo.limts (olso.limits repo, olso.limits spec & oslo.limits etherpad). The keystone team already created a keystone-limits-api that is currently experimental, and the feeling in the room was that we should try and implement it using a new oslo library to find where changes need to be made.

The migration procedure was discussed, and how we (the services) would need to run multiple quota systems for quite a few cycles, due to partial upgrades that happen in OpenStack. [1] Possible implementations were discussed, and the oslo.limits etherpad has a good overview of them.


This is an idea that I have been very interested in since it was discussed in Sydney. We actually had 3 sessions on this in Dublin, across 3 different cross project rooms - API-SIG, Oslo and Self Healing SIG.

Overall, most people were receptive - the commentary is that the spec is too wordy, and contains my favorite description:

It feels like OpenStack, but not really in a good way.

After listening to feedback, and talking offline to a few people I think I have a handle on where the issues are, and I think I have a rough spec I can work flesh out over the next few days. I think I will just start writing code at that point as well - I think with a more concrete example it could help clear up issues for people.

Edge Computing

I stopped by the Edge room on the Tuesday to listen in on the discussion. Personally, I really think this group needs to stop bikesheding on, well, everything, and actually go and implement a POC and see what breaks.

The push still seems to be "make OpenStack work on the edge" instead of (what I think is the quickest / most productive way forward) "write extra tooling to orchestrate OpenStack on the edge."

There was some interesting items brought up, like Glance, and image / data residency. I think that actually engaging with the Glance team might have been helpful, as they were completely unaware that the discussion was being held, but the concepts raised sounded interesting.

I lasted about an hour or so, before I gave up. From my limited exposure, it sounded exactly like the discussions I have heard on the Edge Calls, which were the same as the ones I heard in Sydney.

Designate Sessions

The Designate room was a quiet enough affair, but it marks the first time since the PTG's started that getting a dedicated room was justified. We did some onboarding with new folks, and laid out a plan for the cycle.

The plan so far looks like this:


    • A starting point, using signing keys in the Designate database, which we can use as a jumping point to storing keys in a HSM / Barbican
    • People are currently looking at PowerDNS's inline signing as a short term solution.
  • Docs (it will never not be a priority :) )

  • Shared Zones

  • Improve the UI

    • This really relies on us either rm -rf openstack/designate-dashboard or finding people who understand Angular.

TC / QA / Tempest / Trademark Programs

If you follow the mailing list, or openstack/governance reviews, you may have seen a long running discussion over where tempest tests used for Trademark Programs should go. From memory I seem to remember this being raised in Boston, but it could have been Barcelona. There was tension between QA, me, the InterOp Work Group, and others about the location. Chris Dent covered this pretty well in his TC updates over the last while, so I am not going to rehash it, but it does look like we finally have some sort of agreement on the location, after what was 2 other times I thought we had agreement :).

Board of Directors Meeting

The OpenStack Board of Directors met on the first day of the PTG. This was an issue in its own right, which was highlighted in a thread on the foundation list. Thankfully, I have been told that this situation will not happen again (I can't seem to find any record of the decision, so it may have been an informal board discussion, but if anyone from the board is reading, replying to the foundation list would be great.).

As it met on day one, I didn't get to see much - I arrived to wait for the Add On Trademark program approvals, and happened to catch a very interesting presentation by Dims, Chris and Melvin. I then got to see the board approve DNS as a Trademark add on, which is great for the project, and people who want a constant DNSaaS experience across multiple clouds.

Johnathon Price's Board Recap is also a good overview, with links to things that were presented at the meeting.

The Board, Community, and how we interact

One topic that was highlighted by the TC / QA / Tempest / Trademark Programs discussion was that the QA team is very under resourced. This, combined with the board discussing future of the PTGs due to cost makes me very worried.

The foundation has (in my mind) two main focuses.

  1. Promote the use of the software or IP produced by people working on projects under the foundation, and protect its reputation.
  2. Empower the developers working on said software or IP to do so.

In my eyes, Trademark programs are very much part of #1, and the board should either:

  1. Fund / find resources for the QA team, to ensure they have enough bandwidth to maintain all trademark programs, the associated tests, and tooling.
  2. Fund / find a team that does it separately, but removes the entire burden from the QA team.

The PTG falls firmly under #2. I was initially a PTG skeptic, but I really think it works as an event, and adds much more value than the old mid-cycles did. I understand it has problems, but without it, teams will go back to the mid cycles, which may have looked cheaper at first glance, but for some people either meant multiple trips, or missing discussions.

One very disappointing thing to see was the list of Travel Support Program donors - there was some very generous individuals in the community that stood up and donated, but none of the corporate foundation members contributed. This, with members being added to the foundation that seem to stop at paying the membership fee (see tencent who were added at the Sydney board meeting), makes me wonder about the value placed on the community by the board.

I know the OpenStack Foundation is diversifying its portfolio of projects beyond just the OpenStack Project (this is going to get confusing :/), but we should still be supporting the community that currently exists.

Other great write ups

This turned into a bit of a PTG + 2 weeks after update, so here are some other write ups I have read over the last week or so, and prompted me to remember things that I would have otherwise forgotten.

The Hotel

And, I saved the best until last. The Croke Park Hotel was absolutely amazing during the conference. When we needed to leave the main venue on Thursday, they managed the transition of a few hundred developers into all the public spaces we could find extremely well. They kept us fed, watered and happy the entire time we were in the hotel. The fact they managed to do this, while not leaving the hotel to go home and sleep themselves! I cannot say enough good things about them, and encourage anyone who is looking for a hotel in Dublin to stay there, or is running an event to use Croke Park and the hotel.

[1] I have heard of companies running Newton / Ocata Designate (and other projects) on clouds as old as Liberty.

Sydney OpenStack Summit


OpenStack Down Under

This year the travelling circus that is the OpenStack summit migrated to Sydney. A lot of us in Europe / North America found out exactly how far away from our normal venues it really is. (#openstacksummit on twitter for the days before the summit was an entertaining read :) )

Sunday Board / Joint Leadership Meeting

As I was in Sydney, and staying across the road from the meeting, I decided to drop in and listen. It was an interesting discussion, with a couple of highlights.

Chris Dent had a very interesting item about developer satisfaction - he has blogged about it on his blog: anticdent.org and it is well worth the read.

Johnathon Bryce lead the presentation of a proposed new expansion of the foundation, which he touched on in the Keynote the next day - I have a few concerns, but they are all much longer term issues, and may just be my own interal biases. I think the first new addition to the foundation will let us know how the rest of the process is going to go.

Colleen Murphy and Julia Kreger told us that they (along with Flavio Percoco) will be starting research to help improve our inclusiveness in the community.

The last item was brought forward by 2 board members, and they focused on LTS (Long Term Support / Stable) branches. The time from an upstream release until a user has it in production is actually long than expected - with a lot of time being used by distros packaging and ensuring installers are up to date.

This means that by the time users have a release in production, the upstream branches may be fully deprecated. There was a follow up Forum Session, and there is now an effort to co-ordinate a new methodology for long term collaboration in the LTS Etherpad.

There seems to be an assumption that distros are keeping actual git branches around for the longer term, and not layering patches inside of deb / rpm files, which I think is much more likely. I hope this effort succeeds, but my cynical side thinks this is more of a "fix it for us" cry, than "help us fix it". I suppose we will see if people show up.

One slide from this section was not discussed but concerned me. It was talking about having an enforced "TC Roadmap" which had lines from various workgroups and SIGs. Coming from a project that gets a lot of "Can you do x feature?" (to which I usually respond with "Do you have anyone to write the code?") this concerns me. I understand that it can be hard to get things changed in OpenStack, really I do, but a top down enforced "Roadmap" is not the way forward. Honestly, that two board members of an Open Source foundation think it is is worrying.


Designate had 3 sessions in Sydney:

  • Our project update
  • Project On Boarding
  • Ops Feedback

The project update was good - much improved from Boston, where the 2 presenters were not paid to work on the project. We covered the major potential features, where we were for Cycle goals (both Queens goals completed, and Pike goals underway).

Project on boarding was not hugely attended, but I am hoping that was a side effect of the summit being both smaller and far away.


Ops feedback was great - we got a lot of bugs that were impacting our users and deployers, and collected it in our Feedback Etherpad (any comments welcome).

Cross Project Work

I went to quite a few cross project sessions - there was a good amount of discussion, and some useful work came out of it.

Application Tokens

This is something that had completely slipped past me until now, but the ideas were great, and it would have made things I have done in previous companies much much easier.

Healthchecks per service

We came to a good agreement on how we can do standardised health checks across OpenStack, we now need to write a spec and start coding a new piece of middleware :)

Edge Computing

Not so sure this was worth a vist - it was much more crowded than any of the other Forum sessions I went to, and ended up Bike Shedding on where the Edge ends (we literally spent 10 mins talking about if a car was part of the Edge or a thing managed by the edge.)

I kept hearing "smaller and lighter OpenStack" in that session, but have yet to hear what is too heavy about what we currently have. Nearly all our service scale down to some extent, and you can run a complete infrastructure on an 8GB VM.

Overall, it was a good summit - not too busy, and short. Looking forward to not traveling for the next PTG, I think the DUB -> DOH -> SYD and back drained the enthusiasm for flights for the next few months.

OpenStack TC Nomination

I would like to submit my candidacy for the Technical Committee for the upcoming election.


TL;DR; So, I bit the bullet and ran for the TC :)

I have been contributing to OpenStack since the Havana cycle [1] mainly in Designate. I have also sporadically gotten involved with the TC, and its meetings since Designate applied for incubation all the way back in Atlanta.

I have been PTL for Designate for Mitaka, Newton, Ocata and the Queens cycle, and a core for a longer period. I was also PTL for the Global Load Balancing before it was an unfortunate early casualty of the recent reshuffling within sponsoring organizations in the community.

As part of previous projects, I was both a developer and a heavy user of OpenStack. As part of contributing to the Kubernetes OpenStack integration we ran into a lot of the problems that impact our users, and people who try to integrate with us.

I believe that we all ready have a great base structure in place to help OpenStack evolve, and part of that is too have a group of people from different companies, backgrounds, genders and cultures to drive the project in the Technical Committee.

I believe my experience working in a younger, smaller project within OpenStack is a benefit, along with the experience of working on software as an end user of OpenStack I can help us ensure the Technical Committee is mindful of the unique challenges these projects and users can face.

I have not traditionally been shy about broaching these topics in the past [2] [3] and [4], but I feel it is time I started follow through, and help guide the resolution for these questions, and I now have an employeer who is supportive of me spending more time in the community.

I do really like this community, and I want us to grow, expand and evolve the software we write, without changing what we stand for.

Thank you for taking the time to read this.

  • Graham Hayes (mugsie)
[1] http://stackalytics.com/?release=all&metric=commits&user_id=grahamhayes
[2] http://lists.openstack.org/pipermail/openstack-dev/2016-July/099285.html
[3] http://graham.hayes.ie/posts/openstack-designate-where-we-are/
[4] https://review.openstack.org/#/c/312267/ (and related discussion)

New Adventures

Today marks the start of a new chapter for me - I started as an employee of SUSE this morning. After 3.5 years (I had no idea it was that long) in HPE the time came, and me, and the rest of the team I work with, along with the Cloud Foundry team, and the OpenStack teams all moved en-mass to the new company.

I am really excited to work with SUSE, they really get open source, and do it well. I think with the progression of IaaS / CaaS / PaaS in recent years, we will end up with something like the linux kernel, and a few distros built around core open source components like OpenStack, Kubernetes and Cloud Foundry (or OpenShift).

SUSE has already shown they know how to do an enterprise distribution of an open source product, and I am looking forward to seeing how we do it for the future of compute.

There is a feeling of wistfulness leaving HP(E) - we did some amazing things there, and we had an amazing group of teams working closely to produce the best products we could. Running services in the public cloud will always be a highlight for me, and the level of freedom we were given to work upstream allowed us to have great projects, not just for HPE customers but for the community.

But alas, life is like a river, and the flow has sped up, carrying me into the next adventure!

  • Graham

OpenStack Designate - Where we are.

I have been asked a few times recently "What is the state of the Designate project?", "How is Designate getting on?", and by people who know what is happening "What are you going to do about Designate?".

Needless to say, all of this is depressing to me, and the people that I have worked with for the last number of years to make Designate a truly useful, feature rich project.


TL;DR; for this - Designate is not in a sustainable place.

To start out - Designate has always been a small project. DNS does not have massive cool appeal - its not shiny, pretty, or something you see on the front page of HackerNews (unless it breaks - then oh boy do people become DNS experts).

A line a previous PTL for the project used to use, and I have happily robbed is "DNS is like plumbing, no one cares about it until it breaks, and then you are standing knee deep in $expletive". (As an aside, that was the reason we chose the crocodile as our mascot - its basically a dinosaur, old as dirt, and when it bites it causes some serious complications).

Unfortunately that comes over into the development of DNS products sometimes. DNSaaS is a check box on a tender response, an assumption.

We were lucky in the beginning - we had 2 large(ish) public clouds that needed DNS services, and nothing currently existed in the eco-system, so we got funding for a team from a few sources.

We got a ton done in that period - we moved from a v1 API which was synchronous to a new v2 async API, we massively increased the amount of DNS servers we supported, and added new features.

Unfortunately, this didn't last. Internal priorities within companies sponsoring the development changed, and we started to shed contributors, which happens, however disappointing. Usually when this happens if a project is important enough the community will pick up where the previous group left off.

We have yet to see many (meaningful) commits from the community though. We have some great deployers who will file bugs, and if they can put up patch sets - but they are (incredibly valuable and appreciated) tactical contributions. A project cannot survive on them, and we are no exception.

So where does that leave us? Let have a look at how many actual commits we have had:

Commits per cycle
Havana 172
Icehouse 165
Juno 254
Kilo 340
Liberty 327
Mitaka 246
Newton 299
Ocata 98

Next cycle, we are going to have 2 community goals:

  • Control Plane API endpoints deployment via WSGI
  • Python 3.5 functional testing

We would have been actually OK for the tempest one - we were one of the first external repo based plug-ins with designate-tempest-plugin

For WSGI based APIs, this will be a chunk of work - due to our internal code structure splitting out the API is going to be ... an issue. (and I think it will be harder than most people expect - anyone using olso.service has eventlet imported - I am not sure how that affects running in a WSGI server)

Python 3.5 - I have no idea. We can't even run all our unit tests on python 3.5, so I suspect getting functional testing may be an issue. And, convincing management that re-factoring parts of the code base due to "community goals" or a future potential pay-off can be more difficult than it should.


We now have a situation where the largest "non-core" project [1] in the tent has a tiny number of developers working on it. 42% of deployers are evaluating Designate, so we should see this start to increase.

How did this happen?

Like most situations, there is no single cause.

Certainly there may have been fault on the side of the Designate leadership. We had started out as a small team, and had built a huge amount of trust and respect based on in person interactions over a few years, which meant that there was a fair bit of "tribal knowledge" in the heads of a few people, and that new people had a hard time becoming part of the group.

Also, due to volume of work done by this small group, a lot of users / distros were OK leaving us work - some of us were also running a production designate service during this time, so we knew what we needed to develop, and we had pretty quick feedback when we made a mistake, or caused a bug. All of this resulted in the major development cost being funded by two companies, which left us vulnerable to changes in direction from those companies. Then that shoe dropped. We are now one corporate change of direction from having no cores on the project being paid to work on the project. [2]

Preceding this, the governance of OpenStack changed to the Big Tent While this change was a good thing for the OpenStack project as a whole it had quite a bad impact on us.

Pre Big Tent, you got integrated. This was at least a cycle, where you moved docs to docs.openstack.org, integrated with QA testing tooling, got packaged by Linux distros, and build cross project features.

When this was a selective thing, there was teams available to help with that, docs teams would help with content (and tooling - docs was a mass of XML back then), QA would help with tempest and devstack, horizon would help with panels.

In Big Tent, there just wasn't resources to do this - the scope of the project expansion was huge. However the big tent happened (in my opinion - I have written about this before) before the horizontal / cross project teams were ready. They stuck to covering the "integrated" projects, which was all they could do at the time.

This left us in a position of having to reimplement tooling, figure out what tooling we did have access to, and migrate everything we had on our own. And, as a project that (at our peak level of contribution) only ever had 5% of the number of contributors compared to a project like nova, this put quite a load on our developers. Things like grenade, tempest and horizon plug-ins, took weeks to figure out all of which took time from other vital things like docs, functional tests and getting designate into other tools.

One of the companies who invested in designate had a QE engineer that used to contribute, and I can honestly say that the quality of our testing improved 10 fold during the time he worked with us. Not just from in repo tests, but from standing up full deployment stacks, and trying to break them - we learned a lot about how we could improve things from his expertise.

Which is kind of the point I think. Nobody is amazing at everything. You need people with domain knowledge to work on these areas. If you asked me to do a multi-node grenade job, I would either start drinking, throw my laptop at you or do both.

We still have some of these problems to this day - most of our docs are in a messy pile in docs.openstack.org/developer/designate while we still have a small amount of old functional tests that are not ported from our old non plug-in style.

All of this adds up to make projects like Designate much less attractive to users - we just need to look at the project navigator to see what a bad image potential users get of us. [3] This is for a project that was ran as a full (non beta) service in a public cloud. [4]

Where too now then?

Well, this is where I call out to people who actually use the project - don't jump ship and use something else because of the picture I have painted. We are a dedicated team, who cares about the project. We just need some help.

I know there are large telcos who use Designate. I am sure there is tooling, or docs build up in these companies that could be very useful to the project.

Nearly every commercial OpenStack distro has Designate. Some have had it since the beginning. Again, developers, docs, tooling, testers, anything and everything is welcome. We don't need a massive amount of resources - we are a small ish, stable, project.

We need developers with upstream time allocated, and the budget to go to events like the PTG - for cross project work, and internal designate road map, these events form the core of how we work.

We also need help from cross project teams - the work done by them is brilliant but it can be hard for smaller projects to consume. We have had a lot of progress since the Leveller Playing Field debate, but a lot of work is still optimised for the larger teams who get direct support, or well resourced teams who can dedicate people to the implementation of plugins / code.

As someone I was talking to recently said - AWS is not winning public cloud because of commodity compute (that does help - a lot), but because of the added services that make using the cloud, well, cloud like. OpenStack needs to decide that either it is just compute, or if it wants the eco-system. [5] Designate is far from alone in this.

I am happy to talk to anyone about helping to fill in the needed resources - Designate is a project that started in the very office I am writing this blog post in, and something I want to last.

For a visual this is Designate team in Atlanta, just before we got incubated.


and this was our last mid cycle:


and in Atlanta at the PTG, there will be two of us.

[1] In the Oct-2016 User Survey Designate was deployed in 23% of clouds
[2] I have been lucky to have a management chain that is OK with me spending some time on Designate, and have not asked me to take time off for Summits or Gatherings, but my day job is working on a completely different project.
[3] I do have other issues with the metrics - mainly that we existed before leaving stackforge, and some of the other stats are set so high, that non "core" projects will probably never meet them.
[4] I recently went to an internal training talk, where they were talking about new features in Newton. There was a whole slide about how projects had improved, or gotten worse on these scores. A whole slide. With tables of scores, and I think there may have even been a graph.
[5] Now, I am slightly biased, but I would argue that DNS is needed in commodity compute, but again, that is my view.

Non Candidacy for Designate PTL


Non Candidacy for Designate PTL - Pike

Happy new year!

As you may have guessed from the title, I have decided that the time has come to step aside as PTL for the upcoming cycle. It is unfortunate, but my work has pivoted in a different direction over the last year (containers all the way down man - but hey, I got part of my wish to write Golang, just not on the project I envisaged :) ).

As a result, I have been trying to PTL out of hours for the last cycle and a half. Unfortunatly, this has had a bad impact on this cycle, and I don't think we should repeat the pattern.

We have done some great work over the last year or so - Worker Model, the s/Domain/Zone work, the new dashboard, being one of the first projects to have an external tempest plugin and getting lost in the west of Ireland in the aftermath of the flooding.


I can honestly say, I have enjoyed my entire time with this team, from our first meeting in Austin, back in the beginning of 2014, the whole way through to today. We have always been a small team, but when I think back to what we have produced over the last few years, I am incredibly proud.

Change is healthy, and I have been in a leadership position in Designate longer than most, and no project should rely on a person or persons to continue to exist.

I will stick around on IRC, and still remain a member of the core review team, as a lot of the roadmap is still in the heads of myself and 2 or 3 others, but my main aim will be to document the roadmap in a single place, and not just in thousands of etherpads.

It has been a fun journey - I have gotten to work with some great people, see some amazing places, work on really interestig problems and contribute to a project that was close to my heart.

This is not an easy thing to do, but I think the time is right for the project and me to let someone else make their stamp on the project, and bring it to the next level.

Nominations close soon [0] so please start thinking about if you would like to run or not. If anyone has any questions about the role, please drop me an email [1] or ping me [2] on IRC

Thank you for this opportunity to serve the community for so long, it is not something I will forget.

[0] Election Schedule
[1] graham.hayes (a) hpe.com
[2] mugsie

Equal Opportunities for all OpenStack Projects

So, two weeks ago, I dropped a TC motion and a mailing list post and waited for the other foot to drop.

I was pleasantly surprised - no one started shouting at me - but by trying to not point fingers at individual teams I made the text too convoluted.

So, in an effort to clarify things, here is an overview of what has been said so far, both in the mailing list and the gerrit review itself.


... does this also include plugins within projects, like storage backends in cinder and hypervisor drivers in nova?

No - this was not clear enough. This change is aimed at projects that are points of significant cross project interaction. While, in the future there may come a point where Nova Compute Drivers are developed out of tree (though I doubt it), that is not happening today. As a result, there is no projects in the list of projects that would need to integrate with Nova.

Could you please clarify: do you advocate for a generic plugin interface for every project, or that each project should expose a plugin interface that allows plugin to behave as in-tree components? Because the latter is what happens with Tempest, and I see the former a bit complicated.

For every project that has cross project interaction - tempest is a good example.

For these projects, they should allow all projects in tree (like Nova, Neutron, Cinder etc are today), or they should have a plugin interface (like they currently do), but all projects must use it, and not use parts of tempest that are not exposed in that interface.

This would mean that tempest would move the nova, neutron, etc tests to use the plugin interface.

Now, that plugin could be kept in the tempest repo, and still maintained by the QA team, but should use the same interface as the other plugins that are not in that repository.

Of course, it is not just tempest - an incomplete list looks like:

  • Tempest
  • Devstack
  • Grende
  • Horizon
  • OpenStack Client
  • OpenStack SDK
  • Searchlight
  • Heat
  • Mistral
  • Celiometer
  • Rally
  • Documentation

And I am sure I have missed some obvious ones. (if you see a project missing let me know on the motion)

I think I disagree here. The root cause is being addressed: external tests can use the Tempest plugin interface, and use the API, which is being stabilized. The fact that the Tempest API is partially unstable is a temporary things, due to the origin of the project and the way the scope was redefined, but again it's temporary.

This seems to be the core of a lot of the disagreement - this is only temporary, it will all be fixed in the future, and it should stay this way.

Unfortunately the discrepancy between projects is not temporary. The specific problems I have highlighted in the thread for one of the projects is temporary, but I beleive the only long-term solution is to remove the difference between projects.

Before we start making lots of specific rules about how teams coordinate, I would like to understand the problem those rules are meant to solve, so thank you for providing that example. ... It's not clear yet whether there needs to be a new policy to change the existing intent, or if a discussion just hasn't happened, or if someone simply needs to edit some code.

Unfortunately there is a big push back on editing code to help plugins from some of the projects. Again, having the differing access between projects will continue to exacerbate the problem.

"Change the name of the resolution"

—(Paraphrase from a few people)

That was done in the last patchset. I think the Level Playing Field title bounced around my head from the other resolution that was titled Level Playing Field. It may have been confusing alright.

Other Areas

I feel like I have been picking on tempest a little too much, it just captures the current issues perfectly, and a large number of the community have some knowledge of it, and how it works.

There is other areas across OpenStack the need attention as well:


Horizon privileged projects have access to much more panels than plugins (service status, quotas, overviews etc). Plugins have to rely on tarballs of horizon

OpenStack Client

OpenStack CLI privileged projects have access to more commands, as plugins cannot hook in to them (e.g. quotas)


Plugins may or may not have tempest tests ran (I think that patch merged), they have to use parts of tempest I was told explicitly plugins should not use to get the tests to run at that point.


We can now add install guides and hook into the API Reference, and API guides. This is great - and I am really happy about it. We still have issues trying to integrate with other areas in docs, and most non docs privileged projects end up with massive amounts of users docs in docs.openstack.org/developer/<project> , which is not ideal.

OpenStack - A leveler playing field

I just proposed a review to openstack/governance repo [0] that aims to have everything across OpenStack be plugin based for all cross project interaction, or allow all projects access to the same internal APIs and I wanted to give a bit of background on my motivation, and how it came about.

Coming from a smaller project, I can see issues for new projects, smaller projects, and projects that may not be seen as "important".

As a smaller project trying to fit into cross project initiatives, (and yes, make sure our software looks at least OK in the Project Navigator) the process can be difficult.

A lot of projects / repositories have plugin interfaces, but also have project integrations in tree, that do not follow the plugin interface. This makes it difficult to see what a plugin can, and should do.

When we moved to the big tent, we wanted as a community to move to a flatter model, removing the old integrated status.

Unfortunately we still have areas when some projects are more equal - there is a lingering set of projects who were integrated at the point in time that we moved, and have preferential status.

A lot of the effects are hard to see, and are not insurmountable, but do cause projects to re-invent the wheel.

For example, quotas - there is no way for a project that is not nova, neutron, cinder to hook into the standard CLI, or UI for setting quotas. They can be done as either extra commands (openstack dns quota set --foo bar) or as custom panels, but not the way other quotas get set.

Tempest plugins are another example. Approximately 30 of the 36 current plugins are using resources that are not supposed to be used, and are an unstable interface. Projects in tree in tempest are at a much better position, as any change to the internal API will have to be fixed before the gate merges, but other out of tree plugins are in a place where they can be broken at any point.

None of this is meant to single out projects, or teams. A lot of the projects that are in this situation have inordinate amounts of work placed on them by the big-tent, and I can emphasize with why things are this way. These were the examples that currently stick out in my mind, and I think we have come to a point where we need to make a change as a community.

By moving to a "plugins for all" model, these issues are reduced. It undoubtedly will cause more, but it is closer to our goal of Recognizing all our community is part of OpenStack, and differentiate projects by tags.

This won't be a change that happens tomorrow, next week, or even next cycle, but think as a goal, we should start moving in this direction as soon as we can, and start building momentum.


This was originally posted to the openstack-dev mailing list.