Clearing space on Juju web application units

When deploying to one of our Mojo/Juju-based websites (probably using Jenkins) you may get a Nagios error about there not being enough disk space:

2019-04-03 11:45:38 [ERROR] 1 FAIL on webapp/2:
+ /usr/lib/nagios/plugins/check_disk -u GB -w 25% -c 20% -K 5% -p /
DISK WARNING - free space: / 2 GB (25% inode=66%);| /=6GB;6;7;0;9
...
2019-04-03 11:46:12 [ERROR] 1 FAIL on webapp/3:
+ /usr/lib/nagios/plugins/check_disk -u GB -w 25% -c 20% -K 5% -p /
DISK WARNING - free space: / 2 GB (25% inode=66%);| /=6GB;6;7;0;9

Note down the names of the units that are failing - in this case it’s webapp/2 and webapp/3.

On staging servers, webteam members should have access to fix this ourselves by accessing the role account on wendigo.canonical.com. Let’s use the www.staging.ubuntu.com role account stg-comms-ubuntu-com, and from there SSH onto one of the units and become root (let’s start with webapp/2):

$ ssh {your-canonical-systems-username}@wendigo.canonical.com  # SSH onto wendigo
you@wendigo.com:~$ sudo su - stg-comms-ubuntu-com              # Switch to the correct role account
stg-comms-ubuntu-com@wendigo:~$ juju ssh webapp/2              # Juju SSH onto an application unit
ubuntu@juju-stg-comms-ubuntu-com-machine-5:~$ sudo -s          # Switch to root user

Now we could inspect the disk space with df -h, or try du -sh * to inspect which folders are taking up all the space, but in almost all cases it’s taken up by previous builds, so let’s go ahead and remove those:

# change to the app code directory (replace "webapp-2" as necessary)
$ cd /srv/webapp-2/code/

# Remove all archives
$ rm -r archives/*

# Find which code directory is the "latest"
$ ls -l latest
lrwxrwxrwx 1 wsgi_user wsgi_group 26 Apr  3 10:49 latest -> /srv/webapp-2/code/ee5887e

# Now delete all code directories apart from the one "latest" is pointing to
# NB: Make sure to change "ee5887e" for your latest code directory
$ find . -mindepth 1 -maxdepth 1 -type d ! -name 'archives' ! -name 'ee5887e' -exec rm -r {} \;

# Check all that's left is these few files:
$ ls -l
total 8
drwxr-xr-x 93 wsgi_user wsgi_group 4096 Apr  3 10:49 archives
lrwxrwxrwx  1 wsgi_user wsgi_group   25 Oct  9 08:08 current -> /srv/webapp-2/code/latest
drwxr-xr-x 10 wsgi_user wsgi_group 4096 Apr  3 12:46 ee5887e
lrwxrwxrwx  1 wsgi_user wsgi_group   26 Apr  3 10:49 latest -> /srv/webapp-2/code/ee5887e

# Exit the unit back to stg-comms-ubuntu-com@wendigo
$ exit  # Quit the root account
$ exit  # Exit the unit

Now you should do the same for any other application units that were full - in this case webapp/3, e.g.:

stg-comms-ubuntu-com@wendigo:~$ juju ssh webapp/2
ubuntu@juju-stg-comms-ubuntu-com-machine-5:~$ sudo -s
$ cd /srv/webapp-3/code/
$ rm -r archives/*
$ ls -l latest
$ find . -mindepth 1 -maxdepth 1 -type d ! -name 'archives' ! -name 'ee5887e' -exec rm -r {} \;
$ ls -l