[Spec] git-ubuntu staged uploads

[rbasak]

Index	US048
Title	git-ubuntu staged uploads
Status	Pending Review
Author	Robie Basak
Type	Standard
Created	2020-03-12

Abstract

Currently, a merge proposal of a branch against the git-ubuntu repository for an Ubuntu package is tightly coupled to the upload of the branch after it is approved. But uploading a package triggers additional process which isn’t always desirable. Ubuntu developers would like to stage branches without uploading them immediately. Other Ubuntu developers could then build upon these branches and upload from them in the future when appropriate.

Rationale

Uploading incurs a cost of going through proposed migration. Tests on reverse dependencies may take a long time. Decoupling the upload itself and approval of changes to the package allows us to save the cost of unnecessary uploads.

If we can incorporate a reliable method of staging uploads into git-ubuntu workflows, then:

1. developers whose workflows primarily use git-ubuntu will benefit from this functionality; and

2. some developers who currently do not use any VCS for Ubuntu package maintenance will benefit from starting to do so.

Specification

Namespace and content

The git ref namespace in git-ubuntu imported repositories shall set aside refs/heads/next/* for Ubuntu uploaders to stage changes for future upload; for the purpose of this specification, these shall be called “staging branches”. This subdivision of the ref namespace shall be used as follows:

• refs/heads/next/ubuntu/devel shall be used to stage changes to the development release.

• refs/heads/next/ubuntu/<series>-proposed shall be used to stage changes to stable releases, but only when all such staged changes are suitable for the SRU process and related process (eg. SRU documentation in bugs) are ready.

• Other names inside refs/heads/next/* remain reserved.

The staging branches shall be “unapplied”; staged “applied” branches are not considered and are not in scope of this specification.

Management

The git-ubuntu importer shall fast-forward the staging branches against the non-next unapplied branches when it updates a repository against the latest source package publications in Launchpad. Branches that cannot be fast-forwarded shall be left as-is for Ubuntu developers to resolve the situation manually. Staging branches that do not exist shall be created and pointed to the corresponding importer branches.

Once implemented, we will re-run the importer against every repository. After that, it can be assumed that the staging branches always exist.

Access Control

Only an Ubuntu developer who can upload to a particular package and series shall have permission to push to the corresponding staging branch. Non-fast-forwarding pushes (ie. “force push”) shall also be permitted to those who can push. Pushes to all other refs remain reserved for the importer only.

Team owner rename

Since the ~git-ubuntu-import Launchpad team will now include branches that contain primary work that are not imports, this team shall be renamed to ~git-ubuntu.

Social requirements

These requirements cannot be enforced by implementation of this specification itself, but assumptions are documented here as they are required for the staging system to work in practice. They will need to be socialised and consensus established by public discourse.

• An Ubuntu developer who chooses to push a particular change to a staging branch shall take responsibility for that change as if they uploaded it themselves.

• Uploads to the staging branch for a stable release should only be made for the purpose of staging SRUs and only after associated SRU process requirements (eg. SRU documentation in bugs) have been prepared.

• Ubuntu developers who upload any changes to a package to Ubuntu should endeavour to incorporate any changes previously staged by other Ubuntu developers. However, such incorporation may not always happen. In this case, it will be up to those who had previously staged changes to rebase them and/or upload themselves as required.

Further Information

Design Choices

Location and behaviour of staging branches

Development workflows in Ubuntu are a consequence of Debian and Ubuntu engineering history. Prevailing software engineering practice today is to use a VCS (in our field today, git) as the single source of truth.

Developers, including non-distribution developers, expect there to be a branch (“main”, formerly “master”) that represents the cutting edge and the base against which changes should generally be proposed first.

git-ubuntu currently reflects this principle by presenting its ubuntu/devel branches as equivalent, but this only mirrors the existing status quo of package uploads being the single source of truth and does not provide any view of staged branches in git. To an outside observer, branches used elsewhere for this purpose are not discoverable, and where Vcs-Git is used to point to team-maintained packaging, these aren’t consistent across packages and generally don’t share the commit graph with git-ubuntu.

Ideally, best practice distribution development workflows in Ubuntu would be the same as best practice software development workflows in general, except where the nature of distribution development requires it to be different. A useful test for this is to consider to what extent it is possible to explain our workflows to non-distribution developers such that our ways seem reasonable without using history as an excuse.

git-ubuntu staged branches should therefore be presented in as close a way as possible as how an outside developer would expect to see them.

For now, Ubuntu must accept uploads made directly with dput and without git-ubuntu because so many of our existing workflows expect this. It isn’t practical to force this to change all at once. And Ubuntu developers must be able to confidently identify what is part of the official record (ie. what is actually published as a source package) from the git-ubuntu view. With staging, a consequence of these requirements is that the cutting edge may now appear in one of two places: ubuntu/devel (especially if an upload has been made outside git-ubuntu), or the staging branch. In the case that a direct upload trumps the staging branch, the two branches will be diverged and this will need to be resolved by a maintainer of the staging branch manually.

It’s therefore unavoidably necessary to accommodate in our design the case that either of ubuntu/devel and the staging branch may be the current development tip. The design of the git-ubuntu CLI may be able to help with this, but regardless, the simplest developer experience would be one where both of these branches exist in the same repository and therefore on the same git remote from the user’s perspective.

Using a different repository and making use of Launchpad’s namespacing, such as ~ubuntu-uploaders/ubuntu/+source/<package> was the original plan, but the plan was changed given the above considerations. There were also a few other concerns noted with this alternative:

• When submitting an MP, Launchpad will choose the target repository automatically. This won’t work if it were ~ubuntu-uploaders.

• It’s not possible for third parties to submit MPs against ~ubuntu-uploaders.

• The default repository target is used for object sharing.

• When going to code.launchpad.net in the web UI, a user might expect to see the staging branch in the same repository as “everything else”. But if the staging branch belongs to a different team, this won’t be possible.

• The staging branch wouldn’t be able to be presented in the repository that is default for the package.

Advantages of staging branches being always present:

• Easy (future) “git ubuntu clone” workflow: we can simply always start from pkg/next/ubuntu/devel, the local branch will be called next/ubuntu/devel by default, and tooling will default to push back to there (for those with permission).

• MPs can be filed against the next branch since it exists. They will then be marked automatically for merge.

• Since uploaders will be able to push to next branches, they will eventually be able to manipulate merge proposals, which they cannot currently do with git-ubuntu’s read-only import branches. See LP: #2007731. However this isn’t implemented inside Launchpad yet.

Disadvantages of always being present

• We will need to run the importer against every repository to get these created, and every new repository and new series will need this branch updated, too. However, new repositories and new series already prompt importer runs, and in the common case that the branch pointer is identical to the imported branch, the overhead will be minimal.

Importer behaviour and HEAD

In the common case that nothing is staged, the staging branches should be fast-forwardable in the case that someone uploads without git-ubuntu, since they would move with the corresponding “official” branch pointer. Then we could always point HEAD to the staging branches instead. Repointing HEAD is planned in the future but but out of scope of this spec.

Drawbacks

If a non-git-ubuntu upload occurs and the staging branch becomes non fast-forwardable, this may not be obvious to an Ubuntu developer. A risk is that they will not realise and revert the non-git-ubuntu upload by accident. This can be mitigated by a warning from the CLI, and may be prevented by version number collisions in some cases. But it is possible that a developer will not use these tools and a version number collision will not occur, so the error will arise. Before staging branches, this was unlikely since git-ubuntu developers typically use the “devel” tip which does usually fast-forward. Non fast-forwards are possible, but a non-git-ubuntu upload does not usually cause one.

Events on new release opening

When a new development release opens, next/ubuntu/devel would get carried forwards automatically, which is what developers would expect. The importer will create next/ubuntu/<stable>-proposed when it next runs, which in practice happens when all packages are copied forward to the new release as part of the release opening process. This triggers importer runs for everything because they are new source package publications that are treated as requiring importer runs.

Social Responsibility

Uploads into the archive must be performed by Ubuntu developers who have been individually approved by the Developer Membership Board. This requirement is enforced by ACLs in Launchpad that are managed by the DMB. An upload might have been prepared by the uploading Ubuntu developer themselves; it is also common for an approved uploader to “sponsor” someone else’s work after having reviewed it.

Socially, we expect the approved uploader who signed an upload to take final responsibility for that upload. This may involve additional work if delegating it was not successful, such as ensuring that proposed migration succeeds and dealing with any regressions or other mistakes or consequences.

By decoupling the preparation of branches with their future upload, there are implications for this mechanism of social responsibility. An uploader can’t reasonably be expected to take responsibility for branches that they did not themselves prepare or review. Requiring uploaders to review branches already approved and staged by others for future upload would seem wasteful. If at the point of upload the uploader reviewed and wanted review changes, then that would seem to defeat the point of having staged the branch for upload in the first place. And if they could not request review changes, then we’re back to it being unreasonable for the uploader to take responsibility for that part of the upload.

It therefore follows that the only reasonable way to maintain social responsibility for changes made to the Ubuntu archive is to move that responsibility to those approving pushes to the staged branches, instead of responsibility resting with the person who eventually performs the upload itself.

This requires the uploader to have confidence that all changes previously staged for upload have been staged by somebody who can carry that responsibility—it must be someone approved by the DMB to be able to upload to that particular source package for that particular series and pocket.

The most straightforward way to ensure this is to limit, by ACL, the ability to push to a given staging branch to the same set of people who can upload to the corresponding package, series and pocket. This will require support in Launchpad for git repository ACLs to be tied to upload permissions.

We will allow uploaders to force push to the staging branches since in the case of being trumped by a direct upload, uploaders may wish to rebase the staging branch to resolve the issue.

Tracing the responsible person

For this principle of social responsibility to work, It’s necessary to be able to determine the responsible person. In the case of the staging branches, this translates to determining the person who pushed a particular branch, as distinct from the person who authored the change or performed the commit. But git itself does not record this information.

Together with the Launchpad team, we propose that Launchpad will maintain and publish an audit trail that will record this information.

Dependencies on other work

Launchpad’s git repository ACLs

1. Launchpad’s git ACL design incorporates consideration for permissions based on package uploaders, but this was not yet implemented. Support for this feature is tracked in LP: #1993290. Implementation of this spec is blocked on implementation of this inside Launchpad.

2. Currently, git-ubuntu will need to maintain the same ACL entries for every single repository (currently there are 44k). It would be more convenient if we could set the ACL once to apply to all repositories owned by the team. Support for this feature is tracked in LP: #1993303. We may want to consider implementation of this spec to be blocked on implementation of this inside Launchpad.

3. The ability to determine who pushed to a staging branch is necessary for the social responsibility principle to work. The proposal for this mechanism is tracked in LP: #2017975 but this design hasn’t been committed to yet. The design itself should treated as blocking the implementation of this spec. However the implementation of this feature in Launchpad doesn’t necessarily block progress since, given that push will be gated by an ACL, we can rely on trust of a relatively small set of people in the meantime.

Links

• LP: #1844782: Ubuntu developers cannot stage changes to git-ubuntu uploaders prior to upload

• LP: #1993290: git repository ACLs do not support a “person who can upload this package” type

• LP: #1993303: Cannot set default git ACLs for a team

• LP: #2017975: Cannot trace who pushed a git branch

[schopin]

Thanks a lot for working on this, that makes it much more straightforward to use git-ubuntu on, say, glibc . Overall it sounds good, I just have a slight concern and a suggestion:

• Regarding the shifting of responsibility from upload to push, I like it except that there’s an easy way to identify who uploaded a package, whereas Git doesn’t record who pushes the commits. Even if the code comes through via MPs, the Git history doesn’t have an obvious way to link to it unless the commits contain a LP bug number (assuming the MP itself is linked to the bug).

• In case of divergence, could the importer notify the relevant people? I’m guessing it already has all the needed information.

[rbasak]

Thank you for the excellent feedback!

@schopin

Git doesn’t record who pushes the commits

I think this is really important. Thank you for pointing it out! I asked Colin how we might be able to solve this and his suggestion was that Launchpad could start recording this information and make it available.

I’ve amended the spec and filed LP: #2017975 for consideration by the Launchpad team.

@schopin

could the importer notify the relevant people

This is a great idea! I’d like to defer this to beyond an initial implementation please, just to be able to make incremental progress.

Who exactly should receive the notification? The person who pushed the staging branch? Authors and/or committers of the affected commits in the staging branch? The uploader of the divergence from the staging branch? ubuntu-devel@ perhaps? Some combination of the above?

[schopin]

Agreed on deferring the notification, just wanted to throw it out there. I’d notify whoever pushed last, and whoever caused the divergence (so both the author of the d/changelog entry and the actual sponsor)

[slyon]

Thank you very much for pushing forward a modern VCS based development workflow for Ubuntu!
I very much like the suggested changes.

You mentioned the systemd package above and I’ve worked heavily on that one, so I have a few thoughts on some of the use cases (non-blocking):

• So far we used a shared git history with Debian’s corresponding Salsa repository and used the Debian maintainer’s tags to merge in new version into Ubuntu. Switching that to git-ubuntu we’d need to rebase onto the corresponding pkg/import/DEBIAN_VERSION tags instead, which will consist of mostly synthesized commits… Oftentimes, additional context is given inside the commit message by the Debian maintainer, which might (or might not) be relevant for us. This information would be lost… or rather we’d need to keep a separate Salsa git history to check the relevant commits.
• We’re making heavy use of git-buildpackage, e.g. gbp pq. Some gbp commands can create additional branches, which are mostly local (so not a problem for this spec) and IIUC should not clash with the git-ubuntu namespaces. But it’s something to keep in mind when designing git-ubuntu, to avoid future conflicts between the two tools.
• For bigger changes in systemd we used to create feature branches (e.g. enabling systemd-oomd, dropping i386 binaries, …), which we shared between co-maintainers inside our packaging repository. Such branches might not be ready for “staging”. So I think the idea would be to keep those in our personal Launchpad git namespace and creating merge-proposals against the staging branch. But being in the personal namespace would not allow a co-maintainer to push to my WIP branch. This is something we need to be aware of, it might need additional coordination between co-maintainers. But is probably easy to work around.
• I wonder if we could have some automation or tooling support to adopt the Vcs-Git/Browser fields in debian/control and keep them consistent accross all packages, so that Ubuntu developers can always rely on that information. Right now it’s all over the place. And once we have a common structure for staging branches, it would be good to have that consistently represented in ´d/control. Maybe even (automatically?) adding a XS-Vcs-Git-Ubuntu field in there.

Overall, I think having a common, modern workflow (and git repository location!) across all Ubuntu packages is of much higher priority than the few quirks/adoptions in workflow for existing packaging repos (and their maintainers).

[rbasak]

Thank you for the feedback!

I’d like to address your comments about interactions with Salsa and git-buildpackage more generally. In summary, I don’t think that git-ubuntu is the appropriate tool for that use case, and you should continue with your existing workflow for now.

I designed git-ubuntu as a means of having a unified workflow across all packages because without it we found onboarding difficult. Every package we derive from Debian might have its VCS somewhere different, be laid out differently, need different tooling, and so forth. So just as a new starter got to grips with one workflow, they’d come across something different. So right now, git-ubuntu is really designed around packages being derived from somewhere (via a sync or a package merge) as a stepping stone towards a future where Ubuntu uploads might operate entirely with git.

On the other hand, if the packaging isn’t derived but instead the maintaining team is actually the primary source of the packaging (eg. Ubuntu uploads are made by an Ubuntu developer directly from the same VCS as Debian), then the Debian tooling already works well because that’s what Debian does all the time, and use of git-ubuntu doesn’t really add much, if anything. For example, git-ubuntu currently provides no equivalent of uscan or uupdate or gbp import-orig.

In my spec I mentioned systemd as an example, but I didn’t mean to imply that the team maintaining systemd in Ubuntu should switch to git-ubuntu. I’ll look into adjusting the spec to fix that—probably by finding a more suitable example.

For the time being, I think that if you’re maintaining packaging in Ubuntu in collaboration with Debian VCS, or as a branch of Debian VCS, then you should continue to do that, using whatever tooling is used in Debian (eg. gbp, or dpm, etc).

Similarly, if you’re maintaining a package in Ubuntu that isn’t in Debian, then it’s appropriate to use something like gbp to do that, ignoring git-ubuntu.

On the other hand, git-ubuntu does provide a unified git view onto Ubuntu packages generally, and will continue to do that. It’s nice to point first time contributors to git-ubuntu because it’s unified. So I would like for developers using something other than git-ubuntu to accept merge proposals against their git-ubuntu branches. Doing so should be as simple as a rebase. I’d like for new contributors to have that done for them rather than being asked to change their contribution in a review round-trip. On the other hand I would expect established contributors to learn the team’s non-git-ubuntu workflow though of course.

This isn’t the end of the story—I have future ideas on better integration than this. I tried to write that up here, but it became muddled. But you can look forward to future specs in that direction

[rbasak]

@rbasak

In my spec I mentioned systemd as an example, but I didn’t mean to imply that the team maintaining systemd in Ubuntu should switch to git-ubuntu. I’ll look into adjusting the spec to fix that—probably by finding a more suitable example.

Looking into this deeper, I think there are two possible categories of existing non-git-ubuntu repositories:

1. Like systemd, repositories where Ubuntu’s packaging VCS is derived from Debian’s VCS (usually Salsa). In these cases, I don’t think this staging spec will help for reasons explained in the previous post. Maintainers of these Ubuntu packages are probably best served by continuing what they are doing for now and using Debian’s VCS tooling like gbp or dpm.

2. Theoretically, if a team is using their own non-git-ubuntu VCS for Ubuntu package maintenance that has its own ancestry and doesn’t share common ancestry with Debian VCS, then git-ubuntu staging would benefit them. But I’m not sure that any such team exists.

Therefore I’m no longer sure that developers using non-git-ubuntu VCS maintaining packages in Ubuntu will really benefit from the staging feature.

So who will benefit? It’ll be useful for teams maintaining the majority of Ubuntu packages that don’t currently have an Ubuntu-specific VCS, or who are using git-ubuntu already to maintain Ubuntu packages.

Therefore I think I was mistaken in suggesting systemd at all, the example was misleading, and so I’ll just remove this example from the spec without replacing it.

[cjwatson]

on this from a Launchpad perspective.