How to mirror and manage repositories with the API

Landscape Documentation
9

Hi

i’m trying to sync nvidia repos listed here: https://nvidia.github.io/nvidia-container-runtime/

they are basically https://nvidia.github.io/nvidia-container-runtime/ubuntu22.04/nvidia-container-runtime.list

deb https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/$(ARCH) /
deb https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/$(ARCH) /
(yes, they use 18.04 for all major releases)

Following the steps you suggested i created a distro and added a series

landscape-api create-series --pockets release --components main --architectures amd64 --gpg-key mirror-key --mirror-gpg-key nvidia-gpgkey --mirror-uri https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64/ --mirror-series / jammy libnvidia-container-jammy

but looking at sync status i get:

u’result_text’: u’b"Missing checksums in Release file ‘./lists/update-jammy_%2F_flat_InRelease’!\r\nThere have been errors!\r\n"’,

If i add that repo to a ubuntu jammy machine it works fine, delivering deb packages.

what’s wrong?

Thank you

10

Hi,

It looks like we may have hit a bug: when syncing that repo landscape complains about missing checksums but there is one SHA512 as shown:

root@landscape0001:/var/lib/landscape/landscape-repository/standalone/nvidia-container-runtime-jammy/lists# cat update-jammy_%2F_flat_InRelease
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Architectures: amd64
Codename: bionic
Components: main
Date: Wed, 27 Sep 2017 23:08:53 +0000
Description: NVIDIA container runtime repository
Label: NVIDIA CORPORATION cudatools@nvidia.com
Origin: https://nvidia.github.io/nvidia-container-runtime
Suite: bionic
Version: 1.0
SHA512:
a59ede24b2056ad3c111f61aa2b238c66cff99d3a0fea50193964bfdd1489fccb7b51bdd12003b113caf0582b76325cf3dfba3cd476dbda76b6a8f1765176e7c 38026 Packages
708a3c116b6433b86e6ef186c45db40609a149918e0ece4eeac6fffbbe36e8a2525566de57d3f35e7daf24c5e8d51d98eddd2ac460ef5c19a663ff09c8aae61f 7416 Packages.xz
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEyVsyG2HojBgJxPdZ3crgRPeW7LAFAmDJ3uMACgkQ3crgRPeW
7LAb7g/9FHYd11nCXZObealVnyiGsKpuHYZBcik8P5VjcPMJVEWB2Cd9v3J9tv/b
F81Lz4tIIm21y2dAk3Hcb69CiX2gdd+tX7UIyonH7o8ar6XQpn8HWfdSVkewxYMt
hdmhDHJpMZm53S7sVW/h9+cWTV92rfPVRrf/lkj+MmChIJ7l1JiJq8g9LOK7Ef1G
M5h9E3PWXvVgvverb5JOFq/Veom7y+vMfueVtBCRMReNEM2Rw4Mo0w2vyt5h29qM
qcbessrvE2dHCYiJ5iuD6e6ATLGr4KL6X2xWNhZeaN4ZnUzs8Kc5cwfGpmUIizN/
F0S1zsVGUo/imJWOQXkLtVv8NVsfdtJ+k1xjrVHXthZTIVO5VoGyfBmuskPsL2qO
0d0FrdOI9JUtEmZNd0xKHERh49XMF9cl56iz85xpoxErnBnc5bnmixvZ45ro5Uuc
AuWy9t6PMoN/x1EJOm4e9p7RbCwMIv7qWeWfdh/xYKkeqrtb1Ud/qalZnIjC1Hrf
sR9yeomV5i6g20tfVXqjdPzLxiGhOhxYHFHwnp28Zt+33TlaRwVmRRRKYWIjVnn8
ecnrQSHUW0SwVt9Aj9nVSJcOz8yAolmspeLhNZAKxIcomzLSNngJeeZWmf96EBxf
jdvL1R7WKcoMBhZc1Ht5FoPFZPNL30io2HrJpXOSQOxFHRi1/BI=
=80j7
-----END PGP SIGNATURE-----

1 Like
11

Hello, I was able to replicate this in my own setup following the steps as described. I also get the checksum missing error. I suspect that Landscape may be assuming the presence of sha256 checksums and is not currently configured to use the sha512 checksums. I have opened a bug for this on the Lauchpad.net site. You can follow the progress of the bug there.

1 Like
12

I think one of the repositories tutorial was removed that used to be in https://docs.ubuntu.com/landscape/en/upload-tutorial. We have packages such as corretto java that needs to be uploaded to a custom pocket. Could you bring that document back please?

13

Does this page not contain the information you are looking for? Scroll down to the section which covers “Upload Pockets”

14

I’ve added some custom - non ubuntu mirrors but running into issues with the trying to mirror sync with landscape on microsoft’s ubuntu repos.

landscape-api create-distribution microsoft

landscape-api create-series bionic microsoft
–pockets release
–components main
–architectures amd64
–mirror-gpg-key microsoft-mirror-key
–gpg-key mirror-key
–mirror-uri https://packages.microsoft.com/ubuntu/18.04/prod
–mirror-series bionic

landscape-api sync-mirror-pocket release bionic microsoft

sync results in the failure

./job-handler.log-20230711:Jul 11 14:09:24 job-handler-1 CRIT Unhandled Error\nTraceback (most recent call last):\nFailure: canonical.reprepro.reprepro.RepreproError: reprepro ended with exit code 255 (out=‘b"Calculating packages to get…\r\nCharacter 0x4d not allowed in sourcename: ‘Microsoft’!\r\nTo ignore use --ignore=forbiddenchar.\r\nThere have been errors!\r\n"’, err=‘b’’’)\n

It seems the uppercase M from Microsoft’s reprepro is causing the issue unless my syntax is wrong.
https://packages.microsoft.com/ubuntu/18.04/prod/pool/main/

15

Just to update this status

With the bug fix this allows to get further but because the flat structure specifying the mirror series (ie: bionic, focal, jammy) won’t work.

results in error when trying to sync pocket release

‘result_code’: 255,
‘result_text’: 'b"aptmethod error receiving ’
"‘https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/jammy/InRelease’:\\r\\n’404 "
"Not Found [IP: 152.195.19.142 443]’\r\naptmethod error "
'receiving ’
"‘https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/jammy/Release’:\\r\\n’404 "
"Not Found [IP: 152.195.19.142 443]’\r\naptmethod error "
‘receiving ’
“‘https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/jammy/Release.gpg’:\\r\\n’404 "
"Not Found [IP: 152.195.19.142 443]’\r\nThere have been "
'errors!\r\n”’,
‘schedule_after_time’: None,
‘schedule_before_time’: None,
‘summary’: "Sync pocket ‘release’ of series ‘jammy’ in distribution "
“‘nvidia-cuda-jammy’”,

Is there another way to mirror this flat repo?

16

The error with the Microsoft repository indicates that an --ignore=forbiddenchar configuration can be used to bypass this issue with the capital letters.

To configure this, create a file called options in the conf directory for the repository in question:
/var/lib/landscape/landscape-repository/standalone/microsoft/conf/options

In this file add the ignore option:
ignore forbiddenchar

With this in place, you should be able to run the sync correctly.
landscape-api sync-mirror-pocket release bionic microsoft

I have tested this on Ubuntu Jammy with Landscape 23.03 and this worked for me, so I believe this should work for you as well.

17

There was a bug related to the mirroring of flat repositories that was fixed in Landscape 23.03+13. This appears to not have been released for Jammy at this time, but has been released for Focal.

landscape-server | 23.03+12.1-0landscape0 | landscape-23.03 | jammy | amd64
landscape-server | 23.03+14-0landscape0   | landscape-23.03 | focal | amd64

If you were willing to try the self-hosted Landscape beta, you could install 23.03+17 on Focal, which should contain the fix from 23.03+13 as well.

landscape-server | 23.03+17-0landscape0 | landscape-beta | focal | amd64
landscape-server | 23.03+17-0landscape0 | landscape-beta | jammy | amd64

I have confirmed that this does resolve the issue with the flat repositories on Jammy, and I was able to mirror the nvidia-cuda repository for both Jammy and Focal successfully on the Landscape 23.03+17 version.

If you would rather not use the beta version, you will need to wait for the stable release for Jammy to be updated sometime in the future, to capture that bug fix.

18

I’ve managed to setup 23.03 landscape on a new jammy box, I’m on the last part where I need to pull down the repositories for each flavour of Ubuntu that we support, having some issues, are you able to assist please.

When running the below I get back a landscape login page in json format:

landscape-api --key=XXXXXXXXXXXXXXXXXXXXX --secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX sync-mirror-pocket release bionic ubuntu --json --uri=http://landscape-xxx.xxx.xxxx.net

19

Hello, I have use landscape standalone (onprem) for years. I have a production version that is operating effectively. However, I wanted to standup a test version, and I was successful in doing everything except where I need to CREATE SERIES using LANDSCAPE-API I am running 23.03 LTS on Jammy OS Whenever I try to execute the CREATE-SERIES for bionic I get failure with Status 500, only issue I saw in the logs was an attempt to open an ldap config file DENIED by apparmor landscape profile. I changed the ENFORCE to COMPLAIN and now I see that it is allowed to open the file, but I am still getting the 500 status code. Is there a known issue with what I described and somewhere that explains these Status codes?

21

maybe we can assist each other. what are you doing and what error are you getting?

22

we have a python script we use to pull down the repos and create the series and pockets etc, however this time it’s giving us https timeouts and will not finish pulling down the jobs before erroring.

when trying manually with the command above it will return a login page for landscape in jason format.

I am putting the uri because if I don’t, it will error and ask for the uri, the command on this page does not include the uri

23

First, I recommend setting up the environment variables for your default ubuntu ID that was used when landscape is installed , so you don’t need to specify the keys, secret, api, etc. you c an put this in your .profile so its automatic when you logon:

export LANDSCAPE_API_KEY=“{API access key}”
export LANDSCAPE_API_SECRET=“{API secret key}”
export LANDSCAPE_API_URI=“https://{landscape-hostname}/api/”
export LANDSCAPE_API_SSL_CA_FILE=“{LOCATION OF LANDSCAPE CA CERT”}

Second, I assume the distribution and series were created successfully? Can you confirm?

Third, from command line, what does this give you? landscape-api

Third,

execute curl https://{landscape-hostname}/ what is the result?

execute curl https://{landscape-hostname}/api/ what is the result?

check your /var/log/syslog as well, and see if anything is showing up with respect to landscape-api execution.

now, what is exact response to landscape-api sync-mirror-pocket security bionic ubuntu ?

Do you have other ubuntu OS series installed? focal? Is it the same result?

24

I have added these credentials to .profile, this hasn’t stopped it from prompting me to add these to the command, and using the python script I was able to create the distribution and series, with the landscape API I cannot get that to work either as it will too ask for credentials, uri and then give me a JSON output of:

root@landscape-XXXX-XX:~# landscape-api create-distribution ubuntu --key XXXXXXXXXXXXXX --secret XXXXXXXXXXXXXXXX --uri Index of /ubuntu

Got unexpected server error:

Status: 404

Error message:

404 Not Found

Not Found

The requested URL was not found on this server.

Apache/2.4.29 (Ub

root@landscape-master-a1:~# landscape-api
Landscape API client (Python 3) - version 0.9.0
usage: /usr/local/bin/landscape-api [-h] [–key KEY] [–secret SECRET] [–uri URI] [–json] [–ssl-ca-file SSL_CA_FILE] [action]

Global Arguments:
-h, --help show this help message and exit
–key KEY The Landscape access key to use when making the API request. It defaults to the environment variable LANDSCAPE_API_KEY if not provided.
–secret SECRET The Landscape secret key to use when making the API request. It defaults to the environment variable LANDSCAPE_API_SECRET if not provided.
–uri URI The URI of your Landscape endpoint. It defaults to the environment variable LANDSCAPE_API_URI if not provided.
–json Output directly the JSON structure instead of the Python representation.
–ssl-ca-file SSL_CA_FILE
SSL CA certificate to validate server. If not provided, the SSL certificate provided by the server will be verified with the system CAs. It defaults to the
environment variable LANDSCAPE_API_SSL_CA_FILE if not provided

Actions:
accept-pending-computers
add-access-groups-to-role
add-annotation-to-computers
add-apt-sources-to-repository-profile
add-package-filters-to-pocket
add-permissions-to-role
add-persons-to-role
add-pockets-to-repository-profile
add-tags-to-computers
add-uploader-gpg-keys-to-pocket
approve-activities
associate-alert
associate-package-profile
associate-removal-profile
associate-repository-profile
associate-upgrade-profile
cancel-activities
change-computers-access-group
copy-package-profile
copy-role
copy-script
create-access-group
create-apt-source
create-cloud-otps
create-distribution
create-package-profile
create-pocket
create-removal-profile
create-repository-profile
create-role
create-saved-search
create-script
create-script-attachment
create-series
create-upgrade-profile
derive-series
diff-pull-pocket
disable-administrator
disassociate-alert
disassociate-package-profile
disassociate-removal-profile
disassociate-repository-profile
disassociate-upgrade-profile
edit-package-profile
edit-pocket
edit-removal-profile
edit-repository-profile
edit-saved-search
edit-script
edit-upgrade-profile
execute-script
get-access-groups
get-activities
get-activity-types
get-administrators
get-alert-subscribers
get-alerts
get-apt-sources
get-computers
get-computers-not-upgraded
get-csv-compliance-data
get-distributions
get-event-log
get-gpg-keys
get-juju-environments
get-juju-models
get-not-pinging-computers
get-package-profiles
get-packages
get-pending-computers
get-permissions
get-removal-profiles
get-repository-profiles
get-roles
get-saved-searches
get-script-code
get-scripts
get-settings
get-upgrade-profiles
get-upgraded-computers-by-frequency
get-usn-time-to-fix
import-gpg-key
install-packages
invite-administrator
list-pocket
modify-package-profile
pull-packages-to-pocket
reboot-computers
register-juju-environment
register-juju-model
reject-pending-computers
remove-access-group
remove-access-groups-from-role
remove-annotation-from-computers
remove-apt-source
remove-apt-source-from-repository-profile
remove-apt-sources
remove-apt-sources-from-repository-profile
remove-computers
remove-distribution
remove-gpg-key
remove-juju-environment
remove-juju-model
remove-package-filters-from-pocket
remove-package-profile
remove-packages
remove-packages-from-pocket
remove-permissions-from-role
remove-persons-from-role
remove-pocket
remove-pockets-from-repository-profile
remove-removal-profile
remove-repository-profile
remove-repository-profiles
remove-role
remove-saved-search
remove-script
remove-script-attachment
remove-series
remove-tags-from-computers
remove-upgrade-profile
remove-uploader-gpg-keys-from-pocket
rename-computers
set-settings
shutdown-computers
ssh
subscribe-to-alert
sync-mirror-pocket
unsubscribe-from-alert
upgrade-packages

curl https://{landscape-hostname}/

returns json for landscape login page

curl https://{landscape-hostname}/api

returns Query API Service

/var/log/syslog - doesn’t show any particular errors in regards to landscape API, can only see few entries for when I restarted landscape services

this is the exact response for:

root@landscape-master-a1:/var/log# landscape-api sync-mirror-pocket security bionic ubuntu --key {KEY} --secret {SECRET} --uri Index of /ubuntu

Got unexpected server error:

Status: 404

Error message:

404 Not Found

Not Found

The requested URL was not found on this server.

Apache/2.4.52 (Ubuntu) Server at archive.ubuntu.com Port 80

when trying focal I get the same result

27

when you execute “landscape-api create-distribution ubuntu” that is attempting to create that information on YOUR landscape server. Specifying --uri for the internet URL https://archive/ubuntu.com may cause problems. To minimize issues, Export the variables, and to make sure they are set, execute $echo $variable_name ← do this for each variable. Setting these up first will save you from needing to add anything to the landscape-api commands.

Also, make sure you setup an initial ID on YOUR landscape server. Use your browser and https://(your_server) and register the ID in landscape.

28

I have done as you suggested and exported the variables, and we have setup a user for landscape-api on the server, as well as a user for myself.

after exporting I do not get a prompt for the uri I get more https timeouts:

root@landscape-master-a1:~# landscape-api create-distribution ubuntu

HTTPSConnectionPool(host=‘landscape-int.mol.dmgt.net’, port=443): Max retries exceeded with url: /api/ (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)’)))

the ca file has been exported with:
export LANDSCAPE_API_SSL_CA_FILE=“/path/to/ca/file”

with the landscape_server_ca.crt file specified

openssl s_client -connect landscape-xxx.xxx.xxxx.net:443

when running the above I get a successful handshake

29

How are you executing this? Often I find if you execute within python or other languages the search for the certificate fails because that module/language being used has it’s own CA store somewhere (i.e., its not using the operating system standard path for CA)

30

I am using a linux command to execute this:
landscape-api create-distribution ubuntu

It should be searching that OS standard path for CA?

31

Ok. If using landscape-api at command line then it should find it in /etc/ssl/cert/ if not specifically told where to look , I assume it was placed there. Check the landscape issuing CA cert in there and confirm it is able to verify the landscape cert assigned to your web site. Use openssl command to validate all this.

Also execute update-ca-certificates command, that is sometimes required.

Also, check your apache2 file being used for your server, confirm the certificates are pointing to the same location.