Ubuntu Server team update - 1 July 2019

Hi everyone, below you will find the updates of the Ubuntu Server team members from the last week. If you are interested in discussing a topic please start a thread in the Server area of this Discourse site.

1 Like

Virtualization

  • We work on a bunch of enhancement to spectre mitigations which will make the drawbacks of those mitigations less painful if you run new enough HW/SW/FW. Related work is ongoing with merge reviews, tests and builds in the bugs:
    • 1828495 (x86) arch_capabilities for IceLake/CascadeLake
      On this one @rafaeldtinoco does most of the work - I’m mostly consulting/reviewing, thanks!
    • 1832622 (ppc64el) count cache flush Spectre v2 mitigation (DD 2.3)
  • Ran further tests for Spice and created fixes for issues found to eventually complete (done now) spice 14.2 and related spice-protocol and spice-gtk completed for Eoan.
  • Full cycle of merge/test/review for virt-manager 2.2 which is in Eoan now
  • Started the merge of qemu 4.0 for Eoan. There are plenty of todos that make this more complex than usual, for example plenty of upstream changes hwo to handle machine types. But also two bugs that already need a stack of patches on top of even qemu 4.0 that are planned to be released for Eoan. The basic merge is done and testing has started - a bunch of issues got identified and fixed right now two known issues are left to resolve before considering an upload.

DPDK

  • Apps and automation consuming DPDK seems to have reached the DPDK 18.11 stack that we have in >=Disco. Since upstream deprecated vhost user server mode we ahve dropped some delta which was required to set those up with valid owner/permission to be usable when running unprivileged guests. The new vhost-user-client doesn’t need any of that as qemu provides the sockets. I had some discussions with the OpenVSwitch Charm Author to adapt to that.
  • Prepared and discussed prereqs, but right now still waiting that Debian picks up our merge of rdma-core v24 to continue with DPDK 18.11.2 for Eoan

MIR

  • MIR review lmdb 1833745
  • some updates to the overall context of Mailman3 but no major milestone reached yet. Some extra work to evaluate the feasibility of an alternative to uwsgi.

SRUs

  • qemu completed 1830704 (sske expansion)
  • qemu completed 1829868 (wily type)
  • qemu completed 1830859 (libseccomp issues/Disco)
  • libvirt completed 1830268 (VMX regen cache)
  • Some libvirt and qemu SRUs triggered the automatic crash increase detection, but after analyzing the new cases none of them was related. Although as a positive gimmick it seems the planned split of connection drivers will make it possible to resolve the xen vs virtualbox issue in libvirt (once merged in Debian).

As usual involved in our team internal review and bug triage processes, but out of these all topics big enough to be worth to be mentioned I’m sure others in the team will do so (In that context @ahasenack / @paride - one of you please talk about apache and openssl 1.1.1).

1 Like

Reviews

  • kafka
  • ctdb
  • bind9 merge
  • virt-manager
  • landscape-client

SRU

Other

  • ubuntu-advantage client sporadic testing
  • gvfs testing:
  • squid 4.7 preparation (going ahead of debian)
    • Pushed a branch to Debian to check the upstream signature: MR #8
1 Like

VIRTUALIZATION

  1. QEMU HW mitigations support (ARCH_CAPABILITIES)
    LP: #1828495 | PPA: #1828495 | MERGE: #1828495

Backported ARCH_CAPABILITIES MSR to QEMU 3.1 (Ubuntu Disco, this time). Based on @paelzer’s review on my Bionic patches. PPA is good for testing if anyone is interested.

With this features, guest can now report not being susceptible to a specific side-channel vulnerability. For this particular case, by supporting IA32_ARCH_CAPABILITIES MSR we are able to provide the same MSR to a KVM/QEMU guest, informing its kernel about HW support for:

  • IBRS_ALL (enhanced IBRS support)
  • SKIP_L1DFL_VMENTRY (L1D flush is needed on VMENTRY)
  • RDCL_NO (HW is vulnerable to Rogue Data Cache Load)
  • Foreshadow-NG (OS) vuln. (L1 terminal fault, OS)
  • Foreshadow-NG (VMM) vuln. (L1 terminal fault, VMM)

and making it to take better decisions on which mitigation to use, if one is needed (leveraging guest performance for those CPUs).

  • Libvirt support for {Ice,Cascade}Lake CPUs might be needed now.
  • Still waiting on some more requests from Intel (might be to enable features by default, might be to add Cascade Lake v2, I’ll inform here next week).
  1. Reviewed @paelzer’s QEMU ppc backports on count cache flush Spectre v2 mitigation (CVE) (check his reply for this topic).

UBUNTU HA

  1. Corosync and Redundant Rings - The Totem Protocol Explained
    I published an article I have written not too long ago, explaining how to get redundant rings in corosync and what to expect from configuration option changes.

  2. CTDB NFS HA Enablement
    LP: #722201 | DOC: #722201 | PPA: #722201 | MR: #722201
    Enabling Samba CTDB for NFS HA: @ahasenack provided me good feedback on documentation and how the patch should be organized for a better maintenance. I have made modifications and provided a documentation (in discourse) on how to use this new package (from PPA right now).

1 Like

git repository conversion

Converted to git:

  • UTAH, the automation framework we use for ISO testing
  • qa-jenkins-jobs, the main repository for QA Jenkins jobs definitions
  • jenkins-launchpad-plugin, a Jenkins plugin that triggers job when a new merge proposal is created as well as when it is approved.
  • tarmac, the automatic branch lander for the Bazaar branches

Work in progress:

Performance metrics

Boot speed performance metrics

  • Up and running on all the relevant devices and EC2 cloud instances
  • In development: LXD instances

pycloudlib

  • Pushed a branch with improvements to the instance restart() method and with generic file push/pull methods using SFTP where available

Infrastrucure

  • Monthly maintenance of the server team Jenkins instances
  • Partial redeployment of the Jenkins instances using Ansible

Bugs

  • apache2: triaged #1833039 and setup a test PPA with a tentative fix which unfortunately didn’t work. The fix was eventually driven by @ahasenack.

curtin vmtests

  • Slowly making progress in enabling the vmtests on arm64. More on this next week!
1 Like

cloud-init

curtin

  • [MERGED] Add s390x zkey support: Use hardware encryption if available for dm_crypt devices.
  • [Reviewed] Dan’s branch to remove some vmtest integration skip_by_date decorators.

netplan

  • Reviewed Spec for Administratively down interfaces.
1 Like

Packaging

  • Kafka
    • I’ve been working the past couple months on getting Apache Kafka packaged for Ubuntu Advantage for Apps. Since, like many Java apps, it has quite a number of dependencies, phase I focused on getting a binary build of kafka into a PPA. This was successful, and recent work has involved final polish - cleaning up lintian errors, autopkgtest cases, and getting the package thoroughly reviewed by another set of eyes.
    • This last week I restarted looking into how to do a source package build of Kafka. It’s a bit overwhelming from here at the bottom of the learning curve, but attacking it piecemeal, this past week I focused on just one of the dependencies, zkclient, and worked on figuring out its build issues.
    • Kafka and some of its dependencies uses the Gradle build system (as opposed to Maven), and the binary kafka build needs gradle 5.4.x, which is significantly newer than the 4.4.1 version carried in Ubuntu since 18.04. I’ve placed a copy of the upstream gradle 5.4.1 binary build in a server team PPA, just to pin it down for our own usage. I chatted with the Debian Java folks a bit, and learned they’ve made notable progress recently towards getting a newer gradle into Debian; I look forward to not needing my workarounds in a few months. :slight_smile:
    • For the source building of kafka itself, the main task is getting all of its dependencies packaged. There’s quite a few, so I’ve been writing scripts to automate looking up packaging info, downloading git repositories, storing collected package info into a YAML data file, and so on. (Of course https://xkcd.com/1319/)
    • I arbitrarily picked one of the dependencies, zkclient, and have been digging into its own build issues (gradle, yet again).
  • php 7.2 → 7.3 transition
    • Trial run build of php-geos; went fine
    • php pieces aren’t in git-ubuntu, need to figure out whether they should be added, or handle them manually.
  • fetchmail merge #369322 (6.4.0~beta4-3)
    • dep8 tests
    • first round of review comments
    • waiting on 2nd round review

SRUs

  • librabbitmq SRU (LP: #1790657) - “amqp-tools server parameter unusable”
    • started packaging
  • rabbitmq-server SRU (LP: #1784757). “service breakage during shutdown”
    • started packaging
  • rabbitmq-server SRU (LP: #1773324) - “–version shows %%VSN%%”
    • waiting on feedback from upstream
  • bash SRU (LP: #1822776) - “Spinning CPU on built-in wait”
    • waiting on verification test from reporter

Development

  • usmerges
    • Last week I reimplemented the core code from the Canonical Server team’s Merges Page into a quickie CLI script. This week I added support to it to lookup from not only Debian unstable but also Debian experimental.
    • A couple suggested features I’m planning to add are to 1) differentiate between seeded and non-seeded server team packages (as the former will be higher ‘heat’ for us to get merged), and 2) lookup current upstream versions (maybe using the package’s watches file). The combination of the two features should help flag opportunities where moving ahead of debian may be of value to Ubuntu server users.
  • ppa-dev-tools
    • I dusted off some launchpadlib-toolkit scripts from my X days for creating and using PPAs. I’ve started a rewrite, beginning with the ppa-create script.
1 Like

(Posting a day late because yesterday was Canada Day.)

cloud-init

ubuntu-advantage-client

  • Review of minor fixes in response to an internal UI/UX review
  • A couple of fixes for per-series override processing (#665 and #667)
  • Started internal conversations about how to reduce the number of API calls the client has to make

Miscellaneous

ubuntu-advantage-tools

cloud-init

  • [reviews]
    • Found workaround to allow overriding default user in cloud-init as root and allowing root to ssh with a password LP: #1834676
  • Worked on netplan and systemd-networkd configuration for Azure with @raharper to support multiple-static ips on a dhcp interface for Azure LP: #1815254.
  • hosted and published bi-weekly community status meeting
  • generated and published Ubuntu server team dev-summary
  • Wrote spec for cloud-init rendering netplan configuration on Azure for secondary IPs from IMDS and sent it around to interested Canonical networking and CPC folks