Help on Networking issue on a Multipass Swarm

Multipass
1

Hi Guys,

I’m trying to create a docker swarm with multipass comprised of 2 nodes: the manager and a worker.

I’ve created a private registry on the manager node, referring to it as 172.17.0.1:5000.
I’ve pushed the image to this registry without any issues.

The web-server service listens to the web-server host (listenAndServe Go` function).

When I try to deploy the stack, then I see this log on the web-server service:

> docker service logs web_web-server
2023/11/27 18:15:50 Listening and serving  on https://web-server:8090 in debug mode
2023/11/27 18:15:50 HTTP server error: listen tcp 212.95.74.75:8090: bind: cannot assign requested address

So, the web-server hostname is resolved to 212.95.74.75 in the web-server golang program and it fails.

Below is the compose file

version: '3.8'
services:
  web-server:
    image: 172.17.0.1:5000/web-server
    build:
      context: ..
      dockerfile: ./web/server/Dockerfile
    environment:
      - SERVER_HOST=web-server # HERE I DEFINED THE HOST TO LISTEN TO, PORT IS ALWAYS 8090
    ports:
      - 8090:8090
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]

  web-client:
    image: 172.17.0.1:5000/web-client
    build:
      context: ..
      dockerfile: ./web/client/Dockerfile
    environment:
      - SERVER_HOST=web-server
    command: ['-n', '25000']
    deploy:
      mode: replicated
      replicas: 5
      placement:
        constraints: [node.role == worker]
    depends_on:
      - web-server

If I inspect the manager node network configuration, I can see this IP address 192.168.72.14.

> docker node inspect manager
...
   "Status": {
        "State": "ready",
        "Addr": "192.168.72.14"
    },
    "ManagerStatus": {
        "Leader": true,
        "Reachability": "reachable",
        "Addr": "192.168.72.14:2377"
    }
  }
]

If I thought I couldn’t use the service name, I would probably have used this IP but if I use SERVER_HOST=192.168.72.14 in my compose file, it doesn’t work either.

Tech notes

  • nodes IPs : why do I have 3 IPs per node? I can guess that the swarm has 3 networks: ingress, web_default (created for the purpose of the swarm) and docker_gwbridge.
> multipass ls                                                   
Name                    State             IPv4             Image
manager                 Running           192.168.73.4     Ubuntu 22.04 LTS
                                          172.18.0.1
                                          172.17.0.1
worker                  Running           192.168.73.3     Ubuntu 22.04 LTS
                                          172.17.0.1
                                          172.18.0.1
  • web-server service network conf
> docker service inspect --format '{{ json .Endpoint }}' web-server | jq .
{
  "Spec": {
    "Mode": "vip",
    "Ports": [
      {
        "Protocol": "tcp",
        "TargetPort": 8090,
        "PublishedPort": 8090,
        "PublishMode": "ingress"
      }
    ]
  },
  "Ports": [
    {
      "Protocol": "tcp",
      "TargetPort": 8090,
      "PublishedPort": 8090,
      "PublishMode": "ingress"
    }
  ],
  "VirtualIPs": [
    {
      "NetworkID": "vhaoorbx46167zruk4czdbm10",
      "Addr": "10.0.0.203/24"
    },
    {
      "NetworkID": "p4ahe7agvg1yryd81q1n1ouye",
      "Addr": "10.0.5.2/24"
    }
  ]
}
  • manager node docker-related IPv4
ip -4 -o address show | grep docker
3: docker_gwbridge    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge\       valid_lft forever preferred_lft forever
4: docker0    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0\       valid_lft forever preferred_lft forever
  • docker networks (can’t find any of the network IDs from the web-server network config)
> docker network ls                                                                   
NETWORK ID     NAME              DRIVER    SCOPE
463982250b78   bridge            bridge    local
6c3dbf997c47   docker_gwbridge   bridge    local
p4ahe7agvg1y   web_default   overlay   swarm
d5648dc92f79   host              host      local
vhaoorbx4616   ingress           overlay   swarm
a1ee9f2562f8   none              null      local
1 Like
2

Hi @bigboulard!

Thank you for posting this detailed report. We are by no means Docker experts, so we’ve been doing some research trying to determine what parts (if any) Multipass is playing into this. We will try to reply with more detail soon.

3

Hi @townsend ,

You’re right, but because there’s a docker blueprint available with multipass, I thought I could try to get some help.
I’ve found what are the causes of this issue so, I put it here in case someone else encounters the same one:

  • registry must use the public ipv4 IP: 192.168.72.14 not 172.17.0.1
  • the web-server app must listen to all IPs, i.e 0.0.0.0
  • the certificate generated for running the secured registry on the manager node must be copied in /etc/docker/certs.d/192.168.72.14\5000/ in the worker node to make its docker daemon trust the manager's docker daemon. From here the worker can then pull the image and refer to the service by its name web-server

Best.

1 Like
4

Hi @bigboulard!

Sure, I didn’t mean to insinuate you (or anyone else) shouldn’t ask for help on these types of things. It’s just we may not be very quick on providing help in such cases.

At any rate, thanks for the reply and providing the answer!

Warm regards,
Chris