OnPrem - Landscape using pull pockets fails with "does not have a Release file" and certificate errors

Landscape Documentation
1

We ran into a problem using mirror and pull pockets on our landscape (v23.03) installation.

After setting up our on-prem landscape server using the following commands

landscape-api create-distribution ubuntu

landscape-api create-series --pockets release,updates,security \``--components main,restricted,universe,multiverse --architectures amd64,i386 \``--gpg-key mirror-key --mirror-uri http://archive.ubuntu.com/ubuntu/ \``--mirror-series focal focal ubuntu

landscape-api sync-mirror-pocket release focal ubuntu
landscape-api sync-mirror-pocket updates focal ubuntu
landscape-api sync-mirror-pocket security focal ubuntu

Create staging and production pockets
landscape-api create-pocket --pull-pocket release release-staging focal ubuntu main,universe amd64,i386 pull mirror-key
landscape-api create-pocket --pull-pocket updates updates-staging focal ubuntu main,universe amd64,i386 pull mirror-key
landscape-api create-pocket --pull-pocket security security-staging focal ubuntu main,universe amd64,i386 pull mirror-key
landscape-api create-pocket --pull-pocket release-staging release-production focal ubuntu main,universe amd64,i386 pull mirror-key
landscape-api create-pocket --pull-pocket updates-staging updates-production focal ubuntu main,universe amd64,i386 pull mirror-key
landscape-api create-pocket --pull-pocket security-staging security-production focal ubuntu main,universe amd64,i386 pull mirror-key

Create repository profiles
landscape-api create-repository-profile --description "Staging profile used to evaluate" staging-profile
landscape-api associate-repository-profile --tags staging staging-profile
landscape-api create-repository-profile --description "Production profile used for production system" production-profile
landscape-api associate-repository-profile --tags production production-profile

Associate pockets with profiles
landscape-api add-pockets-to-repository-profile staging-profile release-staging,updates-staging,security-staging focal ubuntu
landscape-api add-pockets-to-repository-profile production-profile release-production,updates-production,security-production focal ubuntu

We then configured the SSL certifcates using certbot

However when we try to update our clients using “apt update” we get the following errors

The certificate chain uses expired certificate. Could not handshake: Error in the certificate verification

While not ideal we got around this issue by removing the verification step via:

touch /etc/apt/apt.conf.d/99verify-peer.conf && echo >>/etc/apt/apt.conf.d/99verify-peer.conf "Acquire { https::Verify-Peer false }

With the certificate issue out of the way we are now seeing the following errors on the clients:

E: The repository ‘http://landscape.X.com/repository/standalone/ubuntu focal-release-production Release’ does not have a Release file.
N: Updating from such a repository can’t be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository ‘http://landscape.X.com/repository/standalone/ubuntu focal-updates-production Release’ does not have a Release file.
N: Updating from such a repository can’t be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository ‘http://landscape.X.com/repository/standalone/ubuntu focal-security-production Release’ does not have a Release file.
N: Updating from such a repository can’t be done securely, and is therefore disabled by default.

We are able to apply updates to the clients, just strange that we are seeing these errors.

On the landscape server we see the following message in the apache access log

[24/May/2023:04:57:16 +0000] “GET /repository/standalone/ubuntu/dists/focal-release-production/InRelease HTTP/1.1” 301 710 “-” “Debian APT-HTTP/1.3 (2.0.2ubuntu0.2) non-interactive”
[24/May/2023:04:57:16 +0000] “GET /repository/standalone/ubuntu/dists/focal-updates-production/InRelease HTTP/1.1” 301 710 “-” “Debian APT-HTTP/1.3 (2.0.2ubuntu0.2) non-interactive”
[24/May/2023:04:57:16 +0000] “GET /repository/standalone/ubuntu/dists/focal-security-production/InRelease HTTP/1.1” 301 712 “-” “Debian APT-HTTP/1.3 (2.0.2ubuntu0.2) non-interactive”
[24/May/2023:04:57:16 +0000] “GET /repository/standalone/ubuntu/dists/focal-release-production/InRelease HTTP/1.1” 404 15691 “-” “Debian APT-HTTP/1.3 (2.0.2ubuntu0.2) non-interactive”
[24/May/2023:04:57:17 +0000] “GET /repository/standalone/ubuntu/dists/focal-release-production/Release HTTP/1.1” 301 706 “-” “Debian APT-HTTP/1.3 (2.0.2ubuntu0.2) non-interactive”
[24/May/2023:04:57:17 +0000] “GET /repository/standalone/ubuntu/dists/focal-updates-production/InRelease HTTP/1.1” 404 11280 “-” “Debian APT-HTTP/1.3 (2.0.2ubuntu0.2) non-interactive”
[24/May/2023:04:57:17 +0000] “GET /repository/standalone/ubuntu/dists/focal-updates-production/Release HTTP/1.1” 301 706 “-” “Debian APT-HTTP/1.3 (2.0.2ubuntu0.2) non-interactive”
[24/May/2023:04:57:17 +0000] “GET /repository/standalone/ubuntu/dists/focal-security-production/InRelease HTTP/1.1” 404 11280 “-” “Debian APT-HTTP/1.3 (2.0.2ubuntu0.2) non-interactive”
[24/May/2023:04:57:17 +0000] “GET /repository/standalone/ubuntu/dists/focal-security-production/Release HTTP/1.1” 301 708 “-” “Debian APT-HTTP/1.3 (2.0.2ubuntu0.2) non-interactive”
[24/May/2023:04:57:17 +0000] “GET /repository/standalone/ubuntu/dists/focal-release-production/Release HTTP/1.1” 404 11280 “-” “Debian APT-HTTP/1.3 (2.0.2ubuntu0.2) non-interactive”
[24/May/2023:04:57:17 +0000] “GET /repository/standalone/ubuntu/dists/focal-updates-production/Release HTTP/1.1” 404 11280 “-” “Debian APT-HTTP/1.3 (2.0.2ubuntu0.2) non-interactive”
[24/May/2023:04:57:18 +0000] “GET /repository/standalone/ubuntu/dists/focal-security-production/Release HTTP/1.1” 404 11280 “-” “Debian APT-HTTP/1.3 (2.0.2ubuntu0.2) non-interactive”

When removing the tag from the affected hosts and dissociating it from the staging / production pull-pockets the error is not present. It appears that pull-pocket are the cause of this issue.

2

The repository in Landscape (quickstart using Apache) is not served over https, without a bunch of additional configurations. This may be configured in a Juju environment using HA-Proxy by default, but I’m less familiar with that. Can you say which setup you are using?

Additionally, given your error messages from apt, they appear to be making requests to http, not https, so I’m unsure why you would have gotten a certificate failure warning. The repository profiles as you appear to be using them will also default to using http for the apt configuration lines, so again I am not sure why you would get a certificate error as these requests should not really be using https in the first place.

Can you also say what version of Ubuntu the clients machines are using? You might need to just update your ca-certificates package to trust your certbot certs (LetsEncrypt?)

sudo apt install ca-certificates

However, if that doesn’t work, you may need to import the root certificate for your server into the trust store on the client machines. To do this copy the root cert in PEM format to /usr/local/share/ca-certificates with a .crt extension and then run:

sudo update-ca-certificates

Beyond that, it looks like maybe there is something in your (apache) virtualhost config that might be redirecting the requests, as I see a lot of 301 responses followed by 404 responses. Are you trying to force a global redirect to https in the (apache) virtualhost configuration on the server? This would also affect the ping request that the clients are making, but we can come back to that later. We’ll stick with your repository issues for now.

If you are doing this global redirect as I suspect, you either need to remove that https redirect, or you need to adjust your virtualhost configuration to allow serving the repository over https as well. You would need to add the following highlighted lines to the https virtualhost config and reload apache to access the repository url over https.

/etc/apache2/sites-enabled/<your-config>.conf

 <VirtualHost *:443>
...
# put this at the end of the block of other aliases in the https vhost
   Alias /repository /var/lib/landscape/landscape-repository

   <Location "/repository">
     Order deny,allow
     Deny from all
     ErrorDocument 403 default
     ErrorDocument 404 default
   </Location>
   <LocationMatch "/repository/[^/]+/[^/]+/(dists|pool)/.*">
     Allow from all
   </LocationMatch>
   <Location "/icons">
     Order allow,deny
     Allow from all
   </Location>

# put this at the end of the block of other rewrites near the end of the file
    RewriteCond %{REQUEST_URI} !^/repository/
...
</VirtualHost>

Reload Apache with the following commands

sudo apache2ctl -t
sudo systemctl reload apache2.service

Let me know if this works or if you can share any other information about your particular setup, and I can try to advise from there.

1 Like
3

Thanks for your help. I think we managed to get it working.

Updating the ca-certificates was required as the landscape client was a clean installation of Ubuntu 20.04 LTS without any patches applied. Updating the ca-certs as described above got us pasted the SSL issues.

You were correct in that the problem was with the apache configuration

On a working installation of Landscape Server (23.03+9) the apache configuration contained:

<VirtualHost *:80>
  ...
  RewriteRule ^/(.*) https://landscape.testbed.local:443/$1 [R=permanent]
</VirtualHost>

While on the problem system it had:

<VirtualHost *:80>
  ...
  RewriteRule ^/(.*) https://landscape.X.com:443/$1 [R=permanent]
  RewriteCond %{SERVER_NAME} =landscape.X.com
  RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

Both of these were quick installed, however it could be that the problem system was a fresh install of 23.03.9 while the working system was upgraded from an earlier version.

After commenting out extra 2 lines we were able to update the clients

1 Like