How to backup and restore Landscape

Documentation Landscape Docs
1

Contents:

Overview

Self-hosted Landscape consists of several stateful components that must be roughly synchronized to guarantee correct functioning of the system as a whole. These include:

  • Landscape Server: At least 6 PostgreSQL databases, a cache of hash databases, and a variety of configuration files.
  • Landscape Client(s): Several SQLite databases for tracking package states.

Landscape Server only supports backup to the latest possible state. So, the only supported backup option is one that permits point-in-time recovery (PITR).

We strongly recommend that administrators of a self-hosted Landscape instance familiarize themselves with PITR facilities and PostgreSQL’s archived logging, also called write-ahead logging (WAL). Some syntactic configuration changes have occurred across PostgreSQL versions, so you should select the documentation for your particular PostgreSQL server version:

Given the wide variety of clients (from physical hardware, to VMs, to containers, some of which may be permanent and others merely temporary), backup of Landscape Clients (if required at all) isn’t covered in this guide.

Note: The database guidelines here don’t apply to juju deployments with Charmed PostgreSQL. Charmed PostgreSQL was introduced in the Landscape 24.04 LTS charm. See Charmed PostgreSQL’s backup and restore documentation for information on backing up and restoring your charmed database.

Define a backup and retention policy

Before configuring your PostgreSQL instance for continuous archiving and PITR, it’s important to decide on a backup policy. You may want to consider the following questions:

  1. When should base backups be taken?
    • Recommended: Daily or weekly depending on volume of WAL logs and desired recovery speed.
  2. How many base backups should be retained at any given time?
    • This dictates the earliest point in time to which you can initially restore.
  3. Where should WAL logs be archived to?
    • Recommended: A separate machine or some form of networked and secure storage.
  4. Where should base backups be stored?
    • Recommended: The same machine as the WAL archive so that all materials necessary for restoration are available in one place.

Although it’s possible to backup and archive on the same machine as the PostgreSQL server, we recommend that you use a separate machine for base backup and archived log storage. This is to allow restoration in case the server becomes inaccessible for any reason. We also recommend that any other files needed to restore the Landscape application server (such as the configuration files listed in a following section) are also copied to this location to allow recovery of the entire service from one location.

Configure PostgreSQL

To configure PostgreSQL:

  1. In the postgresql.conf configuration file, set wal_level, archive_mode, and archive_command according to the PostgreSQL documentation for your server’s version.
  2. Test that your archive_command operates correctly in all circumstances, including returning the correct exit codes.
  3. Once you’re confident the configuration is correct, restart the PostgreSQL service to activate archived logging.
  4. Monitor the archive destination to ensure logs begin to appear there.
  5. (Optional) If your server has very low traffic, you may want to use the archive_timeout setting to force archiving of partial logs after a timeout.
  6. When WAL logs are being archived successfully, construct a script that executes pg_basebackup and stores the result in your base backup storage destination.
  7. Test that this operates correctly as the cluster owner (typically postgres).
  8. Add a cronjob to periodically execute this script (as the cluster owner).

Note that you don’t need to take Landscape offline to perform these backups; pg_basebackup can only execute when the cluster is up. There’s no need to worry about inconsistency between Landscape’s various databases either: a base backup represents the state of the cluster across all databases within it at the instant the backup starts.

Backup server configuration files

The following files should also be copied from your Landscape Server(s) to your backup destination to ensure that restoration of the Landscape application server is also possible:

  • /etc/landscape/*: Landscape configuration files and the on-premises Landscape license
  • /etc/default/landscape-server: Specifies which Landscape application services start on a given machine
  • /etc/apache2/sites-available/<server-name>: The Landscape Apache vhost configuration file, usually named after the FQDN of the server
  • /etc/ssl/certs/<landscape-cert>: The Landscape server X.509 certificate
  • /etc/ssl/private/<landscape-cert>: The Landscape server X.509 key file
  • /etc/postgresql/<pg-version>/main/*: Various PostgreSQL configuration files

You may also want to backup the following log files. They’re not required for normal operation of Landscape Server, but may provide additional information in the case of service outages:

  • /var/log/landscape-server/*: The Landscape Server log files

If any of these files change periodically (e.g., the SSL certificates), you may also want to set up a cronjob to handle backing-up these files regularly.

Test recovery procedures

We recommend that administrators of self-hosted Landscape test their recovery procedures after configuring their Landscape Server(s) for archived logging and PITR. This is to ensure that backups are valid and restorable, and that administrators are familiar with these procedures.

To test the recovery procedures:

  1. Provision a spare server (or servers) and install Landscape Server as you have on your production machine(s)
  2. Stop the Landscape application server, and the PostgreSQL cluster on the spare
  3. Copy configuration files (see prior section) to the spare; you may wish to keep a script handy to perform this task in your backup location
  4. If your spare isn’t installed from scratch (e.g. if it is installed from an image), remove everything under /var/lib/landscape/hash-id-databases
  5. Restore a recent PostgreSQL base-backup onto the spare; this usually involves (re)moving the existing PostgreSQL cluster’s data directory (e.g. /var/lib/postgresql/9.5/main) and replacing it with the contents of the base-backup (or un-tarring the base-backup into it, if tar format was chosen); ensure file ownership and modes are preserved!
  6. Construct an appropriate recovery.conf file in the new PostgreSQL cluster’s data directory; a template for this can be found in /usr/share/postgresql/<pg-version>/main/recovery.conf.sample
  7. Start the PostgreSQL cluster on the spare and watch the PostgreSQL logs to ensure recovery proceeds by retrieving and replaying WAL logs
  8. Once recovery is complete, run /opt/canonical/landscape/scripts/hash-id-databases.sh to regenerate the hash databases cache
  9. Finally, start the Landscape application server on the spare and test it to verify correct operation
1 Like
3

Hi @yanisa-hs ,

As the new postgresql charm 14/stable is in use for the latest charmed Landscape bundles, the currently documented process for backup and restore is no longer valid.

There are two complications with the new charmed Postgres:

  1. There is no local backups yet, but only backup to S3 and restore from S3 are available for the new charmed Postgres.
  2. There is no more passwordless access to Postgres. An attempt to restore backup from one charmed Postgres env into another charmed Postgres env using default operator Postgres admin user, will rewrite operator password and break the environment.

To work around that, I managed to use relation-XX landscape generated Postgres user, which is not getting rewritten during DB restore.

!! Only works with Juju 2 (the workaround here is to use “relation-XX” user and password to Postgres which are stored in relation data for Juju 2, but in Juju 3 those are in secret storage without an obvious way to read them) !!

  1. Create standby environment the same way it described in the old doc, pause landscape-servers there.
  2. Take a backup of the current Postgres
# Create backup
juju run-action --wait postgresql-prod/leader get-password

juju ssh postgresql-prod/leader
pg_dumpall -U operator -p 5432 -h $PGSQL_PRIMARY_IP -f backup.sql
  1. Take a backup of the “standby” postgres, because the charm has generated users and passwords already. We don’t want to lose them. We are taking backup to restore only roles and passwords later. Use the backup procedure from the article above.
# Create backup of roles
juju run-action --wait postgresql-standby/leader get-password

juju ssh postgresql-standby/leader
export PGUSER=operator
export PGPASSWORD=<password>
PGSQL_PRIMARY_IP=

pg_dumpall -U operator -p 5432 -h $PGSQL_PRIMARY_IP -f roles.sql
  1. Restore the backup on your new PostgreSQL. The trick here is that the backup from another postgres will rewrite your roles and you will lose access to the DB in the middle of backup restoration. With the old reactive charm, it could be avoided by using root passwordless access. But there is no passwordless access with the new charm. So we trick it using landscape user from relation data to have admin access to the DB. There is a chance your users in relation data are the same in old and new environments, so check that first:
$ juju show-unit landscape-server/2 | grep "dbname=landscape-server" -A 1
          master: dbname=landscape-server host=10.60.78.63 password=0FoMpCPKfxmGSWCM
            port=5432 user=relation-8

If you have the same relation-8 users for all units on all three environments, then take a step back and ask for help. If there is some user “relation-XX” that only exists on the new env - this is what you need to proceed. Log into the new postgres leader and restore from backup.

juju ssh postgresql-standby/leader
export PGUSER=relation-XX
export PGPASSWORD=<password>
psql -p 5432 -h $PGSQL_PRIMARY_IP -d postgres -f backup.sql
# Then you need to restore "roles.sql" to ensure all the juju generated credentials are in place:
psql -p 5432 -h $PGSQL_PRIMARY_IP -d postgres -f roles.sql
# Verify you have access with the old credentials
export PGUSER=operator
export PGPASSWORD=<password>
psql -p 5432 -h $PGSQL_PRIMARY_IP -d postgres
  1. When you resume landscape-servers on the standby environment, your landscape-server should work.
1 Like