graham.hayes.ie (Posts about openstack)

New Open Infrastructure Project(s)

Graham Hayes — Sat, 20 Apr 2019 00:03:50 GMT

OpenStack Foundation - Board Meeting : 2019/04/08

New OIP (Open Infrastructure Projects)

In a long running program, started in Sydney at the OpenStack Foundation Board meeting, the OpenStack Foundation (OSF) has added the first new project to the Foundation under the OIP program - Kata Containers.

The foundation board has laid out a set of requirements for projects to meet before they are fully "confirmed" into the foundation.

These fall under 5 broad areas:

Focus - does the project do something in an area the board has previously said is an area of interest. Also known as a Strategic Focus Area (SFA)

Governance - Does the project act democratically, and have they had a stable process for this.

Best Practices - does the project follow the lead of other foundation projects with technical practices - CI, code review, documentation, bug management and security processes.

Open-ness - Does the project abide by the four opens.

Activity - How many people use the project, and contribute to it?

Kata Containers

Eric Ernst from the Kata Architectural Committee gave a very in depth overview of Kata and what they are doing in the container space. The slides for this are available on Google Slides.

Focus

Kata strives to solve a large, known problem in the container space, which is a listed SFA previously defined by the board. They started as a pilot project in the foundation, with the merging of Intel and hyper.sh IP.

Many public cloud providers have had to write something like Kata for their operations, and Kata aims to bring this security to the Open Source space.

Governance

The Kata governance is similar to the OpenStack Project governance - they have contributors (ATCs), Maintainers (cores) and an Architecture Committee (TC). There is no PTL role, but as a smaller project, that would be unnecessary.

Best Practices

Kata is in a good place for this - they have CI on each PR, a good set of docs, and a VMT (Vulnerability Management Team) that is very similar to the OpenStack one.

Open-ness

They keep everything in open forums - they use Slack but also bridge the conversations into channels on freenode, so people can use IRC to talk to them, the use open Google Docs for specifications, and allow people to comment on specs to feedback.

They have attended summits before, and reach out to related communities (hypervisors and kubernetes were given as examples).

Activity

This was an area I was initially concerned about - but the names and number of users (in production and R&D / testing) looks like it is gaining a growing community. As is to be expected, a lot of people fall into the "pre prod" / R&D phase - this is a very new technology, with no current enterprise offerings which would make companies start out slow.

After the presentation we moved to questions from the board to the Architecture Committee - I have 2.5 pages of notes from this section, so I will not be talking about every question asked here :)

Questions

How are Kata and OpenStack integrating, and using each other?

Zun is currently looking at integrating Kata as a container runtime, and in the days since the board meeting I have seen mailing list posts on this, and some people have been investigating Magnum as well. Currently there is no real push from the Kata side, as they let end users drive features.

Is there any commercial distribution of Kata?

Currently, no. There is interest from customers, but currently no distro has provided support. SUSE / RedHat / Ubuntu have started to work on support in the linux distros, but not on a product version yet.

Most current users don't want a product version, they are happy to trail blaze themselves.

How does Kata work with the Kubernetes community?

Kata and kubernetes do end to end testing with CRI.O and containerd. As the CRI is the interface, Kata has been (and is) involved in the development of that standard, and has helped drive some of the features.

At the end, Alan Clark proposed the board vote on "Approving confirmation of Kata Containers as a Open Infrastructure Project in the OpenStack Foundation." This was seconded by Imad Sousou. Of the directors present, all bar one voted in favor. Arkady Kanevsky abstained (based on a disagreement on the timing of the vote, not on the substantive motion itself.)

Zuul

Monty Taylor presented on behalf of the Zuul Maintainers. In true four opens style, the presentation was produced on gerrit and zuul itself in the open for the community to comment on as it was being written :).

I took a lot less notes on zuul, as I know the project, and in short it is great. The users are a core part of the governance, and there are multiple large installations outside of the OpenStack CI install. People use it to test everything from kubernetes to network switches, which shows the level of flexibility that is in place in the project.

Questions

Again, I am not going to cover every single question, just the ones I thought were interesting.

Is there a published roadmap?

No - not in a traditional sense. The maintainers have a prioritised list of features, that will get done as they get done (as soon as people show up to write them).

What does the Zuul need from the OpenStack Foundation?

They need help with technical marketing (educating people about how Zuul works, and why it is such a good thing), and outreach. For outreach, Zuul has been seen as an "OpenStack" only thing, and letting people know they can use it without OpenStack would be a good thing for the project.

It was noted there has been cases where people adopted Zuul, and then decided they wanted to bring in OpenStack to help manage the pool of CI VMs.

Licenses - does the board need to pass a specific exception for the GPLv3 sections?

No - the board has already approved the use of GPLv3 in certain areas of the Zuul code base (mainly around ansible integrations, but also in the zuul-preview service). However the board got concerned, and decided to wait until the meeting in Denver to approve Zuul, and the text of any license exception.

By Laws Update

4.16 Open Meetings and Records. Except as necessary to protect attorney-client privilege, sensitive personnel information, discuss the candidacy of potential Gold Member and Platinum Members, and discuss the review and approval of Open Infrastructure Projects, the Board of Directors shall: (i) permit observation of its meetings by Members via remote teleconference or other electronic means, and (ii) publish the Board of Directors minutes and make available to any Member on request other information and records of the Foundation as required by Delaware Corporate Law.

—New OpenStack Foundation By-Laws (change in italics)

At a previous meeting, the board expressed a wish to be able to talk about adding a project in an executive session. The above change was posted to the foundation mailing list a week or so before the meeting.

What this allows for is the board to decide that they want to talk about the new project behind closed doors, where the discussions are not public (people on the phone or in the room have to leave), and the people in the session cannot talk about what was discussed other then in general terms.

I object to having this ability as I think it violates our core principals of the four opens - namely the open governance pillar. For something like this - adding a new member to the family of Open Source Infrastructure projects I think we should stick to the rules we expect these projects to live by.

There was a quote on the agenda that said "Feedback from the community is amenable to the addition w/some requests for word changes", which was unfortunately not quite true.

I had replied to the thread a day or two after it was sent, but it seems most directors do not read the foundation@lists.openstack.org mailing list.

I am still not sure what could be required for an executive session that is not covered by "sensitive personnel information" that would require this.

Personally, for me, this looks like we are not abiding by our own ethos of the Four Opens - I do understand if there is personel issues with a potential project, we would want to have it discussed behind closed doors, but everything else should be in the open. If the project that is about to be included has large enough personnel issues that they could cause issues for its inclusion in the foundation, there is a very high chance that they are going to fail some of the confirmation guidelines, and that is something the community should have visibility into.

Even from an optics perspective - the board deciding to include or not include a project behind closed doors is not something that is representative of the OpenStack community, and not something I think the community should be supporting.

—http://lists.openstack.org/pipermail/foundation/2019-April/002749.html

After sending this there was a discussion in the #openstack-tc channel about the change. This should show that the community is not "generally amenable" to the change.

Personally I cannot see the reason for this change - I do not want to oversimplify this, but if there is not a legal restriction for a director to say why they are for or against a project being included, and the director will only say if they support or do not support a project in an executive session, they need to examine their reasons for being there.

There is a train of thought that we should trust our elected board members - which I do - I remember to trust but verify. If there was a discussion at an executive session, they would not be able to raise a flag to the community that something was amiss.

It is worth noting, that the line about Gold and Platinum members was added as an amendment to the by-laws, and now the default route that a member goes though is:

Presentation from the prospective member.
Executive Session
Vote appears out of the discussion in that session.

Is this what we want to have when we are including new project teams in our community? Or should we do that same thing the that the Technical Committee do when we look at adding a project, and do it all in the open?

So my questions to the directors would be this:

Why do you want to add this to the by-laws? When do you see it being used?

I will be at the board meeting in Denver, and I look forward to hearing the reasoning.

TC Nomination - Rocky Cycle

Graham Hayes — Mon, 16 Apr 2018 14:08:00 GMT

Hi,

I am submitting my candidacy for the OpenStack Technical Committee.

I have been contributing to OpenStack since the Havana cycle [1] [2], mainly in Designate. I have also been involved with the TC, and its meetings since Designate applied for incubation all the way back in Atlanta (the first time we were there).

Over the last 6 months I have become more involved in the TC, and have been an active contributor to TC discussions (both on IRC and in person) and governance [3].

I have been PTL for Designate for Mitaka, Newton, Ocata, Queens and Rocky cycles, and a core for a longer period. I believe my experience working in a younger, smaller project within OpenStack is a benefit. Along with the experience of working on software as an end user of OpenStack I can help us ensure the Technical Committee is mindful of the unique challenges these projects and users can face.

Dublin PTG Summary

Graham Hayes — Thu, 15 Mar 2018 19:23:00 GMT

OpenStack - without the flights

Not getting on a plane was a nice change for an OpenStack event :) - especially as it looks like I would not have made it home for a few days.

Cross Project Days (Monday / Tuesday)

These days are where I think the major value of the PTG is. The cross project days feel like the Summit of old, with more time for topics, and less running around a conference centre trying to cram 2 weeks worth of talks / developer sessions, and other meetings into a few days.

Unified Limits / olso.limits / keystone stored limit data

First up for me was the keystone based limits API for unifying quota data in keystone. It was decided to create oslo.limts (olso.limits repo, olso.limits spec & oslo.limits etherpad). The keystone team already created a keystone-limits-api that is currently experimental, and the feeling in the room was that we should try and implement it using a new oslo library to find where changes need to be made.

The migration procedure was discussed, and how we (the services) would need to run multiple quota systems for quite a few cycles, due to partial upgrades that happen in OpenStack. [1] Possible implementations were discussed, and the oslo.limits etherpad has a good overview of them.

olso.healthcheck

This is an idea that I have been very interested in since it was discussed in Sydney. We actually had 3 sessions on this in Dublin, across 3 different cross project rooms - API-SIG, Oslo and Self Healing SIG.

Overall, most people were receptive - the commentary is that the spec is too wordy, and contains my favorite description:

It feels like OpenStack, but not really in a good way.

After listening to feedback, and talking offline to a few people I think I have a handle on where the issues are, and I think I have a rough spec I can work flesh out over the next few days. I think I will just start writing code at that point as well - I think with a more concrete example it could help clear up issues for people.

Edge Computing

I stopped by the Edge room on the Tuesday to listen in on the discussion. Personally, I really think this group needs to stop bikesheding on, well, everything, and actually go and implement a POC and see what breaks.

The push still seems to be "make OpenStack work on the edge" instead of (what I think is the quickest / most productive way forward) "write extra tooling to orchestrate OpenStack on the edge."

There was some interesting items brought up, like Glance, and image / data residency. I think that actually engaging with the Glance team might have been helpful, as they were completely unaware that the discussion was being held, but the concepts raised sounded interesting.

I lasted about an hour or so, before I gave up. From my limited exposure, it sounded exactly like the discussions I have heard on the Edge Calls, which were the same as the ones I heard in Sydney.

Designate Sessions

The Designate room was a quiet enough affair, but it marks the first time since the PTG's started that getting a dedicated room was justified. We did some onboarding with new folks, and laid out a plan for the cycle.

The plan so far looks like this:

DNSSEC
- A starting point, using signing keys in the Designate database, which we can use as a jumping point to storing keys in a HSM / Barbican
- People are currently looking at PowerDNS's inline signing as a short term solution.
Docs (it will never not be a priority :) )
Shared Zones
Improve the UI
- This really relies on us either rm -rf openstack/designate-dashboard or finding people who understand Angular.

TC / QA / Tempest / Trademark Programs

If you follow the mailing list, or openstack/governance reviews, you may have seen a long running discussion over where tempest tests used for Trademark Programs should go. From memory I seem to remember this being raised in Boston, but it could have been Barcelona. There was tension between QA, me, the InterOp Work Group, and others about the location. Chris Dent covered this pretty well in his TC updates over the last while, so I am not going to rehash it, but it does look like we finally have some sort of agreement on the location, after what was 2 other times I thought we had agreement :).

Board of Directors Meeting

The OpenStack Board of Directors met on the first day of the PTG. This was an issue in its own right, which was highlighted in a thread on the foundation list. Thankfully, I have been told that this situation will not happen again (I can't seem to find any record of the decision, so it may have been an informal board discussion, but if anyone from the board is reading, replying to the foundation list would be great.).

As it met on day one, I didn't get to see much - I arrived to wait for the Add On Trademark program approvals, and happened to catch a very interesting presentation by Dims, Chris and Melvin. I then got to see the board approve DNS as a Trademark add on, which is great for the project, and people who want a constant DNSaaS experience across multiple clouds.

Johnathon Price's Board Recap is also a good overview, with links to things that were presented at the meeting.

The Board, Community, and how we interact

One topic that was highlighted by the TC / QA / Tempest / Trademark Programs discussion was that the QA team is very under resourced. This, combined with the board discussing future of the PTGs due to cost makes me very worried.

The foundation has (in my mind) two main focuses.

Promote the use of the software or IP produced by people working on projects under the foundation, and protect its reputation.
Empower the developers working on said software or IP to do so.

In my eyes, Trademark programs are very much part of #1, and the board should either:

Fund / find resources for the QA team, to ensure they have enough bandwidth to maintain all trademark programs, the associated tests, and tooling.
Fund / find a team that does it separately, but removes the entire burden from the QA team.

The PTG falls firmly under #2. I was initially a PTG skeptic, but I really think it works as an event, and adds much more value than the old mid-cycles did. I understand it has problems, but without it, teams will go back to the mid cycles, which may have looked cheaper at first glance, but for some people either meant multiple trips, or missing discussions.

One very disappointing thing to see was the list of Travel Support Program donors - there was some very generous individuals in the community that stood up and donated, but none of the corporate foundation members contributed. This, with members being added to the foundation that seem to stop at paying the membership fee (see tencent who were added at the Sydney board meeting), makes me wonder about the value placed on the community by the board.

I know the OpenStack Foundation is diversifying its portfolio of projects beyond just the OpenStack Project (this is going to get confusing :/), but we should still be supporting the community that currently exists.

Other great write ups

This turned into a bit of a PTG + 2 weeks after update, so here are some other write ups I have read over the last week or so, and prompted me to remember things that I would have otherwise forgotten.

The Hotel

And, I saved the best until last. The Croke Park Hotel was absolutely amazing during the conference. When we needed to leave the main venue on Thursday, they managed the transition of a few hundred developers into all the public spaces we could find extremely well. They kept us fed, watered and happy the entire time we were in the hotel. The fact they managed to do this, while not leaving the hotel to go home and sleep themselves! I cannot say enough good things about them, and encourage anyone who is looking for a hotel in Dublin to stay there, or is running an event to use Croke Park and the hotel.

Sydney OpenStack Summit

Graham Hayes — Wed, 22 Nov 2017 19:35:33 GMT

OpenStack Down Under

This year the travelling circus that is the OpenStack summit migrated to Sydney. A lot of us in Europe / North America found out exactly how far away from our normal venues it really is. (#openstacksummit on twitter for the days before the summit was an entertaining read :) )

Sunday Board / Joint Leadership Meeting

As I was in Sydney, and staying across the road from the meeting, I decided to drop in and listen. It was an interesting discussion, with a couple of highlights.

Chris Dent had a very interesting item about developer satisfaction - he has blogged about it on his blog: anticdent.org and it is well worth the read.

Johnathon Bryce lead the presentation of a proposed new expansion of the foundation, which he touched on in the Keynote the next day - I have a few concerns, but they are all much longer term issues, and may just be my own interal biases. I think the first new addition to the foundation will let us know how the rest of the process is going to go.

Colleen Murphy and Julia Kreger told us that they (along with Flavio Percoco) will be starting research to help improve our inclusiveness in the community.

The last item was brought forward by 2 board members, and they focused on LTS (Long Term Support / Stable) branches. The time from an upstream release until a user has it in production is actually long than expected - with a lot of time being used by distros packaging and ensuring installers are up to date.

This means that by the time users have a release in production, the upstream branches may be fully deprecated. There was a follow up Forum Session, and there is now an effort to co-ordinate a new methodology for long term collaboration in the LTS Etherpad.

There seems to be an assumption that distros are keeping actual git branches around for the longer term, and not layering patches inside of deb / rpm files, which I think is much more likely. I hope this effort succeeds, but my cynical side thinks this is more of a "fix it for us" cry, than "help us fix it". I suppose we will see if people show up.

One slide from this section was not discussed but concerned me. It was talking about having an enforced "TC Roadmap" which had lines from various workgroups and SIGs. Coming from a project that gets a lot of "Can you do x feature?" (to which I usually respond with "Do you have anyone to write the code?") this concerns me. I understand that it can be hard to get things changed in OpenStack, really I do, but a top down enforced "Roadmap" is not the way forward. Honestly, that two board members of an Open Source foundation think it is is worrying.

Designate

Designate had 3 sessions in Sydney:

Our project update

Project On Boarding

Ops Feedback

The project update was good - much improved from Boston, where the 2 presenters were not paid to work on the project. We covered the major potential features, where we were for Cycle goals (both Queens goals completed, and Pike goals underway).

Project on boarding was not hugely attended, but I am hoping that was a side effect of the summit being both smaller and far away.

Ops feedback was great - we got a lot of bugs that were impacting our users and deployers, and collected it in our Feedback Etherpad (any comments welcome).

Cross Project Work

I went to quite a few cross project sessions - there was a good amount of discussion, and some useful work came out of it.

Application Tokens

This is something that had completely slipped past me until now, but the ideas were great, and it would have made things I have done in previous companies much much easier.

Healthchecks per service

We came to a good agreement on how we can do standardised health checks across OpenStack, we now need to write a spec and start coding a new piece of middleware :)

Edge Computing

Not so sure this was worth a vist - it was much more crowded than any of the other Forum sessions I went to, and ended up Bike Shedding on where the Edge ends (we literally spent 10 mins talking about if a car was part of the Edge or a thing managed by the edge.)

I kept hearing "smaller and lighter OpenStack" in that session, but have yet to hear what is too heavy about what we currently have. Nearly all our service scale down to some extent, and you can run a complete infrastructure on an 8GB VM.

Overall, it was a good summit - not too busy, and short. Looking forward to not traveling for the next PTG, I think the DUB -> DOH -> SYD and back drained the enthusiasm for flights for the next few months.

OpenStack TC Nomination

Graham Hayes — Wed, 11 Oct 2017 13:02:20 GMT

TC Nomination

I would like to submit my candidacy for the Technical Committee for the upcoming election.

I have been contributing to OpenStack since the Havana cycle [1] mainly in Designate. I have also sporadically gotten involved with the TC, and its meetings since Designate applied for incubation all the way back in Atlanta.

I have been PTL for Designate for Mitaka, Newton, Ocata and the Queens cycle, and a core for a longer period. I was also PTL for the Global Load Balancing before it was an unfortunate early casualty of the recent reshuffling within sponsoring organizations in the community.

As part of previous projects, I was both a developer and a heavy user of OpenStack. As part of contributing to the Kubernetes OpenStack integration we ran into a lot of the problems that impact our users, and people who try to integrate with us.

I believe that we all ready have a great base structure in place to help OpenStack evolve, and part of that is too have a group of people from different companies, backgrounds, genders and cultures to drive the project in the Technical Committee.

I believe my experience working in a younger, smaller project within OpenStack is a benefit, along with the experience of working on software as an end user of OpenStack I can help us ensure the Technical Committee is mindful of the unique challenges these projects and users can face.

I have not traditionally been shy about broaching these topics in the past [2] [3] and [4], but I feel it is time I started follow through, and help guide the resolution for these questions, and I now have an employeer who is supportive of me spending more time in the community.

I do really like this community, and I want us to grow, expand and evolve the software we write, without changing what we stand for.

Thank you for taking the time to read this.

Graham Hayes (mugsie)

OpenStack Designate - Where we are.

Graham Hayes — Thu, 09 Feb 2017 18:38:00 GMT

Designate: State of the Unio .. Project

I have been asked a few times recently "What is the state of the Designate project?", "How is Designate getting on?", and by people who know what is happening "What are you going to do about Designate?".

Needless to say, all of this is depressing to me, and the people that I have worked with for the last number of years to make Designate a truly useful, feature rich project.

To start out - Designate has always been a small project. DNS does not have massive cool appeal - its not shiny, pretty, or something you see on the front page of HackerNews (unless it breaks - then oh boy do people become DNS experts).

A line a previous PTL for the project used to use, and I have happily robbed is "DNS is like plumbing, no one cares about it until it breaks, and then you are standing knee deep in $expletive". (As an aside, that was the reason we chose the crocodile as our mascot - its basically a dinosaur, old as dirt, and when it bites it causes some serious complications).

Unfortunately that comes over into the development of DNS products sometimes. DNSaaS is a check box on a tender response, an assumption.

We were lucky in the beginning - we had 2 large(ish) public clouds that needed DNS services, and nothing currently existed in the eco-system, so we got funding for a team from a few sources.

We got a ton done in that period - we moved from a v1 API which was synchronous to a new v2 async API, we massively increased the amount of DNS servers we supported, and added new features.

Unfortunately, this didn't last. Internal priorities within companies sponsoring the development changed, and we started to shed contributors, which happens, however disappointing. Usually when this happens if a project is important enough the community will pick up where the previous group left off.

We have yet to see many (meaningful) commits from the community though. We have some great deployers who will file bugs, and if they can put up patch sets - but they are (incredibly valuable and appreciated) tactical contributions. A project cannot survive on them, and we are no exception.

So where does that leave us? Let have a look at how many actual commits we have had:

Commits per cycle
Havana	172
Icehouse	165
Juno	254
Kilo	340
Liberty	327
Mitaka	246
Newton	299
Ocata	98

Next cycle, we are going to have 2 community goals:

Control Plane API endpoints deployment via WSGI
Python 3.5 functional testing

We would have been actually OK for the tempest one - we were one of the first external repo based plug-ins with designate-tempest-plugin

For WSGI based APIs, this will be a chunk of work - due to our internal code structure splitting out the API is going to be ... an issue. (and I think it will be harder than most people expect - anyone using olso.service has eventlet imported - I am not sure how that affects running in a WSGI server)

Python 3.5 - I have no idea. We can't even run all our unit tests on python 3.5, so I suspect getting functional testing may be an issue. And, convincing management that re-factoring parts of the code base due to "community goals" or a future potential pay-off can be more difficult than it should.

We now have a situation where the largest "non-core" project [1] in the tent has a tiny number of developers working on it. 42% of deployers are evaluating Designate, so we should see this start to increase.

How did this happen?

Like most situations, there is no single cause.

Certainly there may have been fault on the side of the Designate leadership. We had started out as a small team, and had built a huge amount of trust and respect based on in person interactions over a few years, which meant that there was a fair bit of "tribal knowledge" in the heads of a few people, and that new people had a hard time becoming part of the group.

Also, due to volume of work done by this small group, a lot of users / distros were OK leaving us work - some of us were also running a production designate service during this time, so we knew what we needed to develop, and we had pretty quick feedback when we made a mistake, or caused a bug. All of this resulted in the major development cost being funded by two companies, which left us vulnerable to changes in direction from those companies. Then that shoe dropped. We are now one corporate change of direction from having no cores on the project being paid to work on the project. [2]

Preceding this, the governance of OpenStack changed to the Big Tent While this change was a good thing for the OpenStack project as a whole it had quite a bad impact on us.

Pre Big Tent, you got integrated. This was at least a cycle, where you moved docs to docs.openstack.org, integrated with QA testing tooling, got packaged by Linux distros, and build cross project features.

When this was a selective thing, there was teams available to help with that, docs teams would help with content (and tooling - docs was a mass of XML back then), QA would help with tempest and devstack, horizon would help with panels.

In Big Tent, there just wasn't resources to do this - the scope of the project expansion was huge. However the big tent happened (in my opinion - I have written about this before) before the horizontal / cross project teams were ready. They stuck to covering the "integrated" projects, which was all they could do at the time.

This left us in a position of having to reimplement tooling, figure out what tooling we did have access to, and migrate everything we had on our own. And, as a project that (at our peak level of contribution) only ever had 5% of the number of contributors compared to a project like nova, this put quite a load on our developers. Things like grenade, tempest and horizon plug-ins, took weeks to figure out all of which took time from other vital things like docs, functional tests and getting designate into other tools.

One of the companies who invested in designate had a QE engineer that used to contribute, and I can honestly say that the quality of our testing improved 10 fold during the time he worked with us. Not just from in repo tests, but from standing up full deployment stacks, and trying to break them - we learned a lot about how we could improve things from his expertise.

Which is kind of the point I think. Nobody is amazing at everything. You need people with domain knowledge to work on these areas. If you asked me to do a multi-node grenade job, I would either start drinking, throw my laptop at you or do both.

We still have some of these problems to this day - most of our docs are in a messy pile in docs.openstack.org/developer/designate while we still have a small amount of old functional tests that are not ported from our old non plug-in style.

All of this adds up to make projects like Designate much less attractive to users - we just need to look at the project navigator to see what a bad image potential users get of us. [3] This is for a project that was ran as a full (non beta) service in a public cloud. [4]

Where too now then?

Well, this is where I call out to people who actually use the project - don't jump ship and use something else because of the picture I have painted. We are a dedicated team, who cares about the project. We just need some help.

I know there are large telcos who use Designate. I am sure there is tooling, or docs build up in these companies that could be very useful to the project.

Nearly every commercial OpenStack distro has Designate. Some have had it since the beginning. Again, developers, docs, tooling, testers, anything and everything is welcome. We don't need a massive amount of resources - we are a small ish, stable, project.

We need developers with upstream time allocated, and the budget to go to events like the PTG - for cross project work, and internal designate road map, these events form the core of how we work.

We also need help from cross project teams - the work done by them is brilliant but it can be hard for smaller projects to consume. We have had a lot of progress since the Leveller Playing Field debate, but a lot of work is still optimised for the larger teams who get direct support, or well resourced teams who can dedicate people to the implementation of plugins / code.

As someone I was talking to recently said - AWS is not winning public cloud because of commodity compute (that does help - a lot), but because of the added services that make using the cloud, well, cloud like. OpenStack needs to decide that either it is just compute, or if it wants the eco-system. [5] Designate is far from alone in this.

I am happy to talk to anyone about helping to fill in the needed resources - Designate is a project that started in the very office I am writing this blog post in, and something I want to last.

For a visual this is Designate team in Atlanta, just before we got incubated.

and this was our last mid cycle:

and in Atlanta at the PTG, there will be two of us.

Non Candidacy for Designate PTL

Graham Hayes — Mon, 09 Jan 2017 22:16:50 GMT

Non Candidacy for Designate PTL - Pike

Happy new year!

As you may have guessed from the title, I have decided that the time has come to step aside as PTL for the upcoming cycle. It is unfortunate, but my work has pivoted in a different direction over the last year (containers all the way down man - but hey, I got part of my wish to write Golang, just not on the project I envisaged :) ).

As a result, I have been trying to PTL out of hours for the last cycle and a half. Unfortunatly, this has had a bad impact on this cycle, and I don't think we should repeat the pattern.

We have done some great work over the last year or so - Worker Model, the s/Domain/Zone work, the new dashboard, being one of the first projects to have an external tempest plugin and getting lost in the west of Ireland in the aftermath of the flooding.

I can honestly say, I have enjoyed my entire time with this team, from our first meeting in Austin, back in the beginning of 2014, the whole way through to today. We have always been a small team, but when I think back to what we have produced over the last few years, I am incredibly proud.

Change is healthy, and I have been in a leadership position in Designate longer than most, and no project should rely on a person or persons to continue to exist.

I will stick around on IRC, and still remain a member of the core review team, as a lot of the roadmap is still in the heads of myself and 2 or 3 others, but my main aim will be to document the roadmap in a single place, and not just in thousands of etherpads.

It has been a fun journey - I have gotten to work with some great people, see some amazing places, work on really interestig problems and contribute to a project that was close to my heart.

This is not an easy thing to do, but I think the time is right for the project and me to let someone else make their stamp on the project, and bring it to the next level.

Nominations close soon [0] so please start thinking about if you would like to run or not. If anyone has any questions about the role, please drop me an email [1] or ping me [2] on IRC

Thank you for this opportunity to serve the community for so long, it is not something I will forget.

Equal Opportunities for all OpenStack Projects

Graham Hayes — Mon, 25 Jul 2016 21:25:00 GMT

A week or 2 later - where are we on fairness for all projects?

So, two weeks ago, I dropped a TC motion and a mailing list post and waited for the other foot to drop.

I was pleasantly surprised - no one started shouting at me - but by trying to not point fingers at individual teams I made the text too convoluted.

So, in an effort to clarify things, here is an overview of what has been said so far, both in the mailing list and the gerrit review itself.

Feedback

... does this also include plugins within projects, like storage backends in cinder and hypervisor drivers in nova?

No - this was not clear enough. This change is aimed at projects that are points of significant cross project interaction. While, in the future there may come a point where Nova Compute Drivers are developed out of tree (though I doubt it), that is not happening today. As a result, there is no projects in the list of projects that would need to integrate with Nova.

Could you please clarify: do you advocate for a generic plugin interface for every project, or that each project should expose a plugin interface that allows plugin to behave as in-tree components? Because the latter is what happens with Tempest, and I see the former a bit complicated.

For every project that has cross project interaction - tempest is a good example.

For these projects, they should allow all projects in tree (like Nova, Neutron, Cinder etc are today), or they should have a plugin interface (like they currently do), but all projects must use it, and not use parts of tempest that are not exposed in that interface.

This would mean that tempest would move the nova, neutron, etc tests to use the plugin interface.

Now, that plugin could be kept in the tempest repo, and still maintained by the QA team, but should use the same interface as the other plugins that are not in that repository.

Of course, it is not just tempest - an incomplete list looks like:

Tempest
Devstack
Grende
Horizon
OpenStack Client
OpenStack SDK
Searchlight
Heat
Mistral
Celiometer
Rally
Documentation

And I am sure I have missed some obvious ones. (if you see a project missing let me know on the motion)

I think I disagree here. The root cause is being addressed: external tests can use the Tempest plugin interface, and use the API, which is being stabilized. The fact that the Tempest API is partially unstable is a temporary things, due to the origin of the project and the way the scope was redefined, but again it's temporary.

This seems to be the core of a lot of the disagreement - this is only temporary, it will all be fixed in the future, and it should stay this way.

Unfortunately the discrepancy between projects is not temporary. The specific problems I have highlighted in the thread for one of the projects is temporary, but I beleive the only long-term solution is to remove the difference between projects.

Before we start making lots of specific rules about how teams coordinate, I would like to understand the problem those rules are meant to solve, so thank you for providing that example. ... It's not clear yet whether there needs to be a new policy to change the existing intent, or if a discussion just hasn't happened, or if someone simply needs to edit some code.

Unfortunately there is a big push back on editing code to help plugins from some of the projects. Again, having the differing access between projects will continue to exacerbate the problem.

"Change the name of the resolution"

—(Paraphrase from a few people)

That was done in the last patchset. I think the Level Playing Field title bounced around my head from the other resolution that was titled Level Playing Field. It may have been confusing alright.

Other Areas

I feel like I have been picking on tempest a little too much, it just captures the current issues perfectly, and a large number of the community have some knowledge of it, and how it works.

There is other areas across OpenStack the need attention as well:

Horizon

Horizon privileged projects have access to much more panels than plugins (service status, quotas, overviews etc). Plugins have to rely on tarballs of horizon

OpenStack Client

OpenStack CLI privileged projects have access to more commands, as plugins cannot hook in to them (e.g. quotas)

Grenade

Plugins may or may not have tempest tests ran (I think that patch merged), they have to use parts of tempest I was told explicitly plugins should not use to get the tests to run at that point.

Docs

We can now add install guides and hook into the API Reference, and API guides. This is great - and I am really happy about it. We still have issues trying to integrate with other areas in docs, and most non docs privileged projects end up with massive amounts of users docs in docs.openstack.org/developer/<project> , which is not ideal.

OpenStack - A leveler playing field

Graham Hayes — Thu, 14 Jul 2016 19:30:07 GMT

I just proposed a review to openstack/governance repo [0] that aims to have everything across OpenStack be plugin based for all cross project interaction, or allow all projects access to the same internal APIs and I wanted to give a bit of background on my motivation, and how it came about.

Coming from a smaller project, I can see issues for new projects, smaller projects, and projects that may not be seen as "important".

As a smaller project trying to fit into cross project initiatives, (and yes, make sure our software looks at least OK in the Project Navigator) the process can be difficult.

A lot of projects / repositories have plugin interfaces, but also have project integrations in tree, that do not follow the plugin interface. This makes it difficult to see what a plugin can, and should do.

When we moved to the big tent, we wanted as a community to move to a flatter model, removing the old integrated status.

Unfortunately we still have areas when some projects are more equal - there is a lingering set of projects who were integrated at the point in time that we moved, and have preferential status.

A lot of the effects are hard to see, and are not insurmountable, but do cause projects to re-invent the wheel.

For example, quotas - there is no way for a project that is not nova, neutron, cinder to hook into the standard CLI, or UI for setting quotas. They can be done as either extra commands (openstack dns quota set --foo bar) or as custom panels, but not the way other quotas get set.

Tempest plugins are another example. Approximately 30 of the 36 current plugins are using resources that are not supposed to be used, and are an unstable interface. Projects in tree in tempest are at a much better position, as any change to the internal API will have to be fixed before the gate merges, but other out of tree plugins are in a place where they can be broken at any point.

None of this is meant to single out projects, or teams. A lot of the projects that are in this situation have inordinate amounts of work placed on them by the big-tent, and I can emphasize with why things are this way. These were the examples that currently stick out in my mind, and I think we have come to a point where we need to make a change as a community.

By moving to a "plugins for all" model, these issues are reduced. It undoubtedly will cause more, but it is closer to our goal of Recognizing all our community is part of OpenStack, and differentiate projects by tags.

This won't be a change that happens tomorrow, next week, or even next cycle, but think as a goal, we should start moving in this direction as soon as we can, and start building momentum.

Designate Newton Design Summit

Graham Hayes — Thu, 05 May 2016 17:00:00 GMT

I started writing this on a plane on the way home from Austin, TX where we just finished up the Newton design summit for OpenStack, and finished it a few days later - please excuse any wierdness in syntax or flow :)

Austin was its usual weird, wacky, wonderful self. Everytime we are here, I have a great time, I just can't deal with the heat :).

The summit format this year worked really well - I pretty much stayed in the design summit hotel for the week - and got some very good work done.

6 Months On: Where are we

One of the things I did before this design summit was to look at what we had achived last cycle.

We had a quiet cycle overall, but we did merge some vital features.

Operators no longer need to use the awfull config format we created for pools in Kilo, we now have a much easier to read YAML file that is loaded into the database via the designate-manage cli, we now actually support multiple pools in a real way with the addition of a schedular in designate-central.

Where are we going?

So - the point of the design summit is to plan the future of the projects - and designate is no execption. We were in a (very nice boardroom) room for a few hours - and we talked through quite a few things.

The collection of etherpads for the summit are available as well.

Golang Replacement of MiniDNS

One of the nicer things about our current architecture is the flexibility that we have because we use the standard DNS protocols to update the target DNS servers. This has the downside however, that we are writing code that deals with DNS packets in python, which is slow. timsim from RackSpace has writen a POC in go that has a very large perfomance improvement.

This needs to documented, and permission requested from the TC to move this component to Go, (and will be a separate post in its own right).

After we got back it turned out that we were not the only project considering this, and swift actually have a feature branch with code in place. So, based on this, we are going to collaborate with them on the integration of Go to OpenStack.

Worker Model

As we dicussed in Galway - we need to replace the current *-manager services with a generic service that can scale out horizontally. As part of this we planned out our upgrade and implementation plans for these services.

Docs, Deprications and more Docs

Docs were a common theme - we were asked to improve them, and also have them located in the main docs on the OpenStack.org website.

We had a member of the docs team in the room, who gave us some great guidenance on how to include our docs in the correct place.

VMT - The application process

On of the teams that supports OpenStack is the Vunrebility Management Team. They deal with disclosures and assigning OSSA numbers to issues that could present and issue to our deployers.

They had a session this summit on how that process might work, and designate was one of the projects chosen to be used as a pilot as I have previously produced Threat Analysis documentation for Designate inside of Hewlett Packard Enterprise - this information is currently being processed for release to the community.

Searchlight

Searchlight is a new (ish) project in OpenStack that enables true searching capabilities on clouds. We have a designate plugin, but there are issues with how we emit notifications from the v1 and the v2 API.

We decided that when we move the Horizon panels to v2, we will just listen for v2 notifications in Searchlight.

API Docs

There was an interesting session on how the community are moving to document their APIs. It will now reside in the projects repo, and is based on a custom sphinx extention that was written for OpenStack.

As this progresses we will migrate designate to these docs, and remove our current docs, as they are much harder to read.