HAProxy for lxd And ACME Config Issue

irtazahussain · April 25, 2025, 6:35am

I want to put my lxd-host behind HAProxy and want to configure ACME for my lxd-host. I did these configs which are shared below. And Here is the config for my HAProxy. Both HAProxy and lxd-host is running on the same VM. My sub-domain is pointing to my server. I want to use Letsencrypt for certs.

ACME Config:

config:
  acme.agree_tos: "true"
  acme.ca_url: https://acme-v02.api.letsencrypt.org/directory
  acme.domain: lxd.testdotpkdomasdfasdfasfain.com
  acme.email: irtazawani100@gmail.com
  core.https_address: '[::]:8443'

HAProxy Config:

# Global configuration
global
  log /dev/log local0
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin
  stats timeout 30s
  user haproxy
  group haproxy
  daemon
  ssl-default-bind-options ssl-min-ver TLSv1.2
  tune.ssl.default-dh-param 2048
  maxconn 100000

# Default settings
defaults
  mode tcp
  timeout connect 5s
  timeout client 30s
  timeout client-fin 30s
  timeout server 120s
  timeout tunnel 6h
  timeout http-request 5s
  maxconn 80000

# Default backend - Return HTTP 301 (TLS upgrade)
backend http-301
  mode http
  redirect scheme https code 301

# Default backend - Return HTTP 403
backend http-403
  mode http
  http-request deny deny_status 403

# HTTP dispatcher (redirect to HTTPS)
frontend http-dispatcher
  bind :80
  mode http
  # Redirect HTTP to HTTPS
  use_backend http-301

# SNI dispatcher for SSL traffic
frontend sni-dispatcher
  bind :443
  mode tcp

  tcp-request inspect-delay 5s

  # SSL inspection (reject non-TLS traffic)
  tcp-request content reject unless { req.ssl_hello_type 1 }

  default_backend http-403

  # Dispatch traffic based on SNI
  use_backend lxd-nodes if { req.ssl_sni -i lxd.testdotpkdomasdfasdfasfain.com }

# Backend for LXD nodes
backend lxd-nodes
  mode tcp
  option tcp-check

  # Backend is configured with localhost, assuming LXD is running on the same server
  server lxd-node01 localhost:8443 check

Lxc Monitor:

INFO   [2025-04-25T07:31:10Z] http: TLS handshake error from 3.17.154.255:37830: tls: first record does not look like a TLS handshake
DEBUG  [2025-04-25T07:31:28Z] Allowing untrusted GET                        ip="192.168.1.109:49576" url=/1.0
WARNING[2025-04-25T07:31:28Z] Rejecting request from untrusted client       ip="192.168.1.109:49576"
DEBUG  [2025-04-25T07:31:28Z] Allowing untrusted GET                        ip="192.168.1.109:49576" url=/1.0
INFO   [2025-04-25T07:32:22Z] http: TLS handshake error from 3.17.154.255:34768: tls: client offered only unsupported versions: [303 302 301]
DEBUG  [2025-04-25T07:32:22Z] Allowing untrusted GET                        ip="192.168.1.109:49576" url=/1.0
WARNING[2025-04-25T07:32:22Z] Rejecting request from untrusted client       ip="192.168.1.109:49576"
DEBUG  [2025-04-25T07:32:22Z] Allowing untrusted GET                        ip="192.168.1.109:49576" url=/1.0

C:\Users\Hp>curl -I http://lxd.testdotpkdomasdfasdfasfain.com
HTTP/1.1 301 Moved Permanently
content-length: 0
location: https://lxd.testdotpkdomasdfasdfasfain.com/


C:\Users\Hp>curl -I https://lxd.testdotpkdomasdfasdfasfain.com
HTTP/1.1 200 OK
Content-Type: application/json
Date: Fri, 25 Apr 2025 07:23:17 GMT
Content-Length: 114

edlerd · April 25, 2025, 11:44am

I’d like to help you, but not sure what is the issue at hand?

I am not sure about this snippet, I think you can remove it.

I’d recommend raising these two to at least a couple of minutes, or your console/terminal sessions will be killed quite soon after establishing the connection.

irtazahussain · April 25, 2025, 11:55am

I am following this documentation

https://documentation.ubuntu.com/lxd/latest/authentication/

# Global configuration
global
  log /dev/log local0
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin
  stats timeout 30s
  user haproxy
  group haproxy
  daemon
  ssl-default-bind-options ssl-min-ver TLSv1.2
  tune.ssl.default-dh-param 2048
  maxconn 100000

# Default settings
defaults
  mode tcp
  timeout connect 500000
  timeout client 500000
  timeout client-fin 500000
  timeout server 500000
  timeout tunnel 6h
  timeout http-request 500000
  maxconn 80000

# Default backend - Return HTTP 301 (TLS upgrade)
backend http-301
  mode http
  redirect scheme https code 301

# Default backend - Return HTTP 403
backend http-403
  mode http
  http-request deny deny_status 403

# HTTP dispatcher (redirect to HTTPS)
frontend http-dispatcher
  bind :80
  mode http
  # Redirect HTTP to HTTPS
  use_backend http-301

# SNI dispatcher for SSL traffic
frontend sni-dispatcher
  bind :443
  mode tcp

  tcp-request inspect-delay 5s

  default_backend http-403

  # Dispatch traffic based on SNI
  use_backend lxd-nodes if { req.ssl_sni -i lxd.testdotpkdomasdfasdfasfain.com }

Browser Error:

This site can’t provide a secure connection
lxd.testdotpkdomasdfasdfasfain.com sent an invalid response.

Try running Windows Network Diagnostics.
ERR_SSL_PROTOCOL_ERROR

irtazahussain · April 25, 2025, 11:58am

I want to use HAProxy for lxd-host with lets-encrypt certs. I have a domain and it’s pointing to my server-ip. Now I want to configure ACME for lxd with lets-encrypt so that my certificate can be automatically handle by it.

tomp · May 1, 2025, 8:04am

@sdeziel1 any ideas on this one?

sdeziel1 · May 1, 2025, 1:36pm

This message says the client only supports TLS 1.0 up to 1.2. The 303 here is in fact the hexadecimal version 0x0303 which maps to TLS 1.2.

LXD requires TLS 1.3 (0x0304) so could you try and configure HAProxy to offer this version?

I think using this set of global config (taken from Mozilla TLS configurator would do:

global
    # modern configuration
    ssl-default-bind-curves X25519:prime256v1:secp384r1
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options prefer-client-ciphers ssl-min-ver TLSv1.3 no-tls-tickets

    ssl-default-server-curves X25519:prime256v1:secp384r1
    ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-server-options ssl-min-ver TLSv1.3 no-tls-tickets

Please let us know if that works as we’d need to update our docs.

tomp · May 1, 2025, 1:48pm

Thanks for debugging the issue @sdeziel1

It would be great if we can improve that error message, as its far from clear what is going on there.

sdeziel1 · May 1, 2025, 3:34pm

The error in question is not coming from LXD. I suspect it’s from HAproxy (openssl lib).

irtazahussain · May 4, 2025, 10:25am

The first error was related to SSL because I was using an SSL certificate for testdotpkdomasdfasdfasfain.com, which was not a wildcard certificate. I have now created a specific SSL certificate for lxd.testdotpkdomasdfasdfasfain.com and converted it into a PEM format. That resolved the SSL-related error.

However, I am now facing a 403 Forbidden error when I access lxd.testdotpkdomasdfasdfasfain.com in the browser. But when I include the port 8443, it loads LXD perfectly.

Now, I want to solve two issues:

How can I access LXD on port 443 without having to specify 8443?
How can I use the domain’s SSL certificates in my LXD host’s certificate store so that I can access my LXD host securely?

root@incus-testing:~# ll  /etc/haproxy/ssl/
total 12
drwxr-xr-x 2 root root 4096 Apr 28 05:53 ./
drwxr-xr-x 4 root root 4096 May  4 10:20 ../
-rw-r--r-- 1 root root 3198 Apr 28 05:53 passworks.io.pem

defaults
  mode tcp
  timeout connect 5s
  timeout client 30s
  timeout client-fin 30s
  timeout server 120s
  timeout tunnel 6h
  timeout http-request 5s
  maxconn 80000

# Default backend - Return HTTP 301 (TLS upgrade)
backend http-301
  mode http
  redirect scheme https code 301

# Default backend - Return HTTP 403
backend http-403
  mode http
  http-request deny deny_status 403

# HTTP dispatcher (redirect to HTTPS)
frontend http-dispatcher
  bind :80
  mode http
  # Redirect HTTP to HTTPS
  use_backend http-301

# SNI dispatcher for SSL traffic
frontend sni-dispatcher
  bind :443 ssl crt /etc/haproxy/ssl/
  mode tcp

  tcp-request inspect-delay 5s

  default_backend http-403

  # Dispatch traffic based on SNI
  use_backend lxd-nodes if { req.ssl_sni -i lxd.testdotpkdomasdfasdfasfain.com }

# Backend for LXD/Incus nodes
backend lxd-nodes
  mode tcp
  option tcp-check

  # Backend is configured with localhost, assuming LXD is running on the same server
  server lxd-node01 localhost:8443 check