How to use Multipass remotely

This document will demonstrate how to use Multipass remotely. This can be useful, for example, if you want to run your VMs on a more powerful machine.

Contents:

Expose Multipass to your network

To expose Multipass to your network, pass --address when starting the daemon:

$ multipassd --help
# ...
  --address <server_name:port>                      specifies which address to
                                                    use for the multipassd
                                                    service; a socket can be
                                                    specified using
                                                    unix:<socket_file>

The client accepts the MULTIPASS_SERVER_ADDRESS environment variable that overrides the default:

$ MULTIPASS_SERVER_ADDRESS=<hostname>:51001 multipass find
Image                       Aliases           Version          Description
...
21.10                       impish            20220118         Ubuntu 21.10

Caveats:

  • Because mounts are executed as privileged users, it is recommended to use client authentication, so you can explicitly allow clients access with a shared passphrase.
    Alternatively, you can multipass set local.privileged-mounts=false to disable the mounts feature altogether.
  • Additionally, the mount command takes a target filepath which is resolved daemon-side, meaning that directories are mounted from the system that the daemon is running on, not the client.
  • Some commands (shell, exec, mount) currently rely on direct networking between the client and the instance, for those to work you’ll need to ensure routing between them is possible.

Change the daemon settings

Linux

You can systemctl edit snap.multipass.multipassd.service and place content along these lines (replace <hostname> with the hostname or the IP you want it to listen on) in:

[Service]
ExecStart=
ExecStart=/usr/bin/snap run multipass.multipassd --address <hostname>:51005

Restart the service then:

$ snap stop multipass
$ snap start multipass

macOS

On macOS you’ll need to add it to the service definition (again, replace <hostname>) and reload it:

$ sudo /usr/libexec/PlistBuddy \
  -c "Add :ProgramArguments: string --address" \
  -c "Add :ProgramArguments: string <hostname>:51001" \
  /Library/LaunchDaemons/com.canonical.multipassd.plist
$ sudo launchctl unload /Library/LaunchDaemons/com.canonical.multipassd.plist
$ sudo launchctl load /Library/LaunchDaemons/com.canonical.multipassd.plist

NB: reinstallation / upgrade will overwrite those changes

Windows

On Windows you’d need to edit/recreate the service definition, and that’s non-trivial (if you do, remember to pass /svc as the first argument to the multipassd.exe binary, and that we currently use the LocalService account)…

As a one-time thing you can use the Start parameters field in the Services panel after stopping the service:


Let us know how this worked for you and what you’d like to see next!

3 Likes

Looks excellent! One small query. If I am running Multipass on a Linux host, is there a guide on exposing the bridged IPs of those vms to the LAN network? (other than routes on the remote machine) If I’m managing my Multipass VMs from a remote host it is likely that I would need to be able to also access the services on that system remotely (a website etc…).

Hope that makes sense! I tried bridging Multipass to my LAN on a linux machine before (using LXD driver) and it just wouldn’t work. I had to use Windows in the end as it bridged correctly, but would prefer to use Linux obviously. But I could be missing something. Any guide or pointers appreciated!

Cheers

@rootisgod if you’re after exposing the instances on your network, --network is the way to go. Would you please file an issue with some details?

https://github.com/canonical/multipass/issues/new?labels=bug&template=bug_report.md


As for ensuring connectivity between your host and remote instances, that would require setting up routes appropriately on your workstation, which would roughly be ip route add <network> via <multipass host IP>. Assuming that you’re on Linux:

$ MULTIPASS_SERVER_ADDRESS=10.2.0.9:51001 multipass launch
Launched: passionate-macaw

$ multipass shell passionate-macaw              
shell failed: ssh connection failed: 'Timeout connecting to 192.168.66.23'

$ sudo ip route add 192.168.66.0/24 via 10.2.0.9

$ multipass exec passionate-macaw -- uname -a   
Linux passionate-macaw 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 18:07:25 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

On any other host adding the appropriate route should equally work :).

Thanks very much for the reply. Let me test my exact setup again and file an issue if I can’t get a bridge working. Thanks for the route commands, that will definitely go a long way, ta.

@saviq

Can I use localhost (127.0.0.1) or even 0.0.0.0 (for example 0.0.0.0:51001) here ?

Lets say I am running a 24 core/64 GB dedicated server on linode with ubuntu-server 22.04 and multipass installed in it. If I install gui and rdp which runs on port 3389 (inside the multipass instance), how do I connect to that rdp from outside over the internet? Or How do I even connect to multipass instance over ssh ?

[Lets not bother with security here, we can tackle that later]

My local host is MacOS.

I am using Namecheap Subdomain redirection to map a web subdomain name (myApp.MyDomian.com) to my cable internet modem, using Port Forwarding to push thru the modem router (https://mycableModemWebFacingIP:AppPortnumber). The Multipass instance has port AppPortnumber implemented in (ports.conf) and in the Apache site adaptor (in place of 443). [lsof] shows its listening on IPV6. I can ssh and scp into the instance. I am having problems with HTTPS into the instance. It hangs. This may be because Multipass has mapped to IPV6, even though I tried to force IPV4. Two questions: 1. Do I have this set up correctly to expose IP:PORT to HTTP?. 2. Assuming I do, and that my Cable Internet Modem needs IPV4, how Can I force IPV4 IP mapping. Nothing I do seems to help. it’s always maps to IPV6.

Three (3) questions, if I may:

  1. Multipass Network Port Exposure: Do I understand it correctly, that 51001 is default Multipass exposure for remote access to Multipass, vs to a Multipass instance? And as such there is only one port exposure (which I can change with the above config)?

  2. Multipass Instance Network Port Exposure: If that is true, and I do not want to expose Multipass, but only want to expose Multipass Instances, each assigned to a different port, is that covered completely when I add the port to ports.conf, add the additional config to NetPlan, add the port routing to the app site adaptor in apache sites-enabled, and allow ports thru ufw? Is there anything more I need to do from Multipass to gain exposure of the instance to the network?

  3. IPv4 Mapping: In doing the above, It seems to map to IPv6. How do I force it to Maputo IPv4?[quote=“saviq, post:3, topic:26360, full:true”]
    @rootisgod if you’re after exposing the instances on your network, --network is the way to go. Would you please file an issue with some details?

https://github.com/canonical/multipass/issues/new?labels=bug&template=bug_report.md


As for ensuring connectivity between your host and remote instances, that would require setting up routes appropriately on your workstation, which would roughly be ip route add <network> via <multipass host IP>. Assuming that you’re on Linux:

$ MULTIPASS_SERVER_ADDRESS=10.2.0.9:51001 multipass launch
Launched: passionate-macaw

$ multipass shell passionate-macaw              
shell failed: ssh connection failed: 'Timeout connecting to 192.168.66.23'

$ sudo ip route add 192.168.66.0/24 via 10.2.0.9

$ multipass exec passionate-macaw -- uname -a   
Linux passionate-macaw 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 18:07:25 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

On any other host adding the appropriate route should equally work :).
[/quote]

Hi @bainmckay. I couldn’t grasp all of what you said, but I will try to answer the best I can.

The first clarification is that Multipass does not do port forwarding or mapping. What you can do is bridge the instance to the outside with an additional network, but you may need to setup your host/router/firewall to let traffic through.

Do I understand it correctly, that 51001 is default Multipass exposure for remote access to Multipass, vs to a Multipass instance?

Yes. That is the port to reach the Multipass Service, not instances themselves. For that you need the special route in the instructions you quoted.

Is there anything more I need to do from Multipass to gain exposure of the instance to the network?

Yes: launch the instance with the --network which also allows you to reach the host (where the instance is running). Allowing or forwarding traffic via ufw is outside the scope of Multipass though.

IPv4 Mapping: In doing the above, It seems to map to IPv6. How do I force it to Maputo IPv4?

I don’t understand. The instructions you quoted use only IPv4. Do you perhaps need to convince your router (or whatever acts as DHCP server) to hand out IPv4?

1 Like

I noticed something out defaulting to IPV6 and then even when IPV4, it shows IPV6. This may be what’s happening.

I am trying to complete the routing from subdomain ipforwarding through my cable internet router to macOS (Ventura). I use tcpdump to see what the OS can see.

  • I set route on macOS as: sudo route -n add -net cableRouterIP/24 multipassExposedIP. This allows tcpdump to see it in macOS
  • I set route on Ubuntu as: sudo route -n add -net cableRouterIP/24 multipassExposedIP. This allows tcpdump to see it in ubuntu
  • Running the http://subdomainaddress, redirects to cableRouterIP:forwardingPort
    ** tcpdump in MacOs displays the packets
    ** tcpdump in Ubuntu displays the packets
  • Now I need to set NetPlan to route to the exposed port
    ** Key concern is, I have no backup. Once I loose the network because Netplan is incorrect, I need rio trash to VM and start over. it has a lot of apps installed on it.
    *** I used to have a backup plan, where I sudo cd inside the Context container folder, nav to the app name and copy the ISO and IMG to another folder. I can’t get that to work anymore.

What am I missing?

I’ve done a lot more work on this to better isolate why it hangs trying to read the default apache website.

ISOLATION PROCEDURE

I redirect khub7subdomain to my cable internet modem router: https://ROUTER_IP: PORT_FORWARDED to push through the router into MacOS Ventura on a (m1) Mac Studio Ultra, hosting Multipass.

*I test the that the port is open on my cable modem using http://yuogetsignal.com/tools/open-ports/

I created a Multipass Ubuntu 20.04 instance called khub7 with --network option, which created an IP gateway (MULTIPASS_GATEWAY_IP) into the instance.

  • I create a route in MacOS from the Cable Internet Modem router to the Multipass instance IP gateway

** sudo route add -net ROUTER_IP/24 MULTIPASS_GATEWAY_IP

  • I create the matching route in Ubuntu

** sudo route add -net ROUTER_IP netmask 255.255.255.0 gw MULTIPASS_GW_IP

I place the DCV (for SSL validation) of the ROUTER_IP SSL in the site root (/var/www/html/.well-known/pki-validation./dcvfilename.txt) which contains the ssl validation test for the ROUTER_IP CSR of the private key.

I map the site-adaptor, which listens to the port forwarded under mod_ssl, to the private key , the crt, and the ca-bundle SSL files.

I verify that Port Forwarding is happing on both MacOS and Ubuntu

  • MacOS

    • nc -vu ROUTER_IP PORT_FORWARDED
  • Ubuntu

    • nc -vu ROUTER_IP PORT_FORWARDED

Tracking access

  • I create a TCPDUMP session in MacOS

** tcpdump -n -i any port PORT_FORWARDED -v

  • and in Ubuntu

** tcpdump -n -i any port PORT_FORWARDED -v

I enter the https request in the browser: https://khub7subdomian, and use browser Inspector on the network tab to monitor the authentication workflow

  • it validates subdomain access with my dns provider, performs the (301) redirect, and submits https://ROUTER_IP:PORT_FORWARDED in the browser.

  • Both TCP packet trackers list the packets coming into MacOS and into Ubuntu

  • The htp request times out in the browser

in Ubuntu, I test access with CURL

  • curl -v -L https://khub7subdomain

  • it responds

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

  • old SSL session ID is stale, removing

  • Connection state changed (MAX_CONCURRENT_STREAMS == 100)!

< HTTP/2 301

HTTP/2 301

< content-type: text/html

content-type: text/html

< content-length: 707

content-length: 707

< date: Tue, 05 Sep 2023 19:01:59 GMT

date: Tue, 05 Sep 2023 19:01:59 GMT

< server: LiteSpeed

server: LiteSpeed

< location: [https://](https://ROUTER_IP:PORT_FORWARDED

location: [https://](https://ROUTER_IP:PORT_FORWARDED

< x-turbo-charged-by: LiteSpeed

x-turbo-charged-by: LiteSpeed

  • Ignoring the response-body

  • Connection #0 to host khubsubrdomain left intact

  • Clear auth, redirects to port from 443 to PORT_FORWARDED Issue another request to this URL: '[https://](https://ROUTER_IP:PORT_FORWARDED

  • Trying ROUTER_IP:PORT_FORWARDED…

  • TCP_NODELAY set

And hangs. I don’t know where to go from here. Is this a Multipass issue or have i missed something?

I should add:

  • localhost works: CURL http://localhost
  • For SSL, of course, I need to use https://ROUTER_IP:PORT_FORWARDED, which is the test I did. Directly or by Redirect. There should e no difference.

Is this the bug where Multipass hangs on redirect, or has that been fixed?

If it is that bug, does switching to the VirtualBox driver help?

I solved the issue above.

You need to have 2 IPs. One (192.xxx.xxx.xxx) is the gateway from the Cable Internet Modem, to map the port forwarded to the Multipass Instance name, and the other IP (10.xxx.xxx.xxx) does the work in the Instance.

  • So the launch has --network en0 and --network en1

    • multipass launch focal --name MyAppInstanceName- --cpus 4 --disk 80G --memory 8G --network en0 --network en1
  • The Instance name enumerates in the Cable Modem app (e.g. Rogers-Shaw home connect, as an extension of the set top device registration tech for TVs etc) using the 192.xxx.xxx.xxx Mac address. You assign a port number for the forwarding.

    • You can place more than one URL-addressable app in the same Multipass instance simply by creating a subdomain for each app, mapping to specified app port number in the site adaptor redirects, assigning multiple ports to the Multipass VM device name in the Cable Internet Router Modem, and listening to the assigned port number in app site adaptor.

      • This includes CSS and JS et all references, but it’s light and fast so no TCP traffic bog down/thrashing that I can see.
    • You add the port forwarded number to ports.conf in mod_ssl and the site ssl adaptor (sites-enabled) listens to the port for TCP SSL traffic.

    • Then the SSL in the instance maps to a subdomain IP SSL Certificate (whose domain name is the IP address of your cable modem).

  • So, Authentication Workflow is a two step process.

    • The first step is authenticating the subdomain SSL certificate, which is registered on your Domain Hosting provider’s site. In my case NameCheap.

      • You need to collaborate with them because SSLs don’t like IP numbers for domain names.

        • You generate the CSR on your instance, use that to generate the SSLCert.

        • Then that SSL certificate goes into each Multipass instance behind your Cable Internet Modem.

        • As well, you need to include the DCV text file (from the CSR SSL issued) in the .well-known/pkivalidation folder in the site root folder.

          • And, to validate your Cable Router, you will need to do a port 80 pass thru to a temporary site adaptor for the first instancer so that the cable modem endoint gets registered as a known site in the domain name hosting provider subdomian folder.
        • So the first authentication is to the subdomain SSL on your Domain Hosting provider (namecheap).

        • It issues a redirect to the cable internet modem IP with forwarded port( eg: redirect: https://70.344.23.15:5024).

        • The redirect passes through your cable internet modem across the Multipass Instance gateway, and is picked up by the site-adaptor (listening to the port number).

        • The site adaptor maps to the cable internet modem IP SSL certificate .

        • Upon authentication, is passes the request to your app site root.

        • The .htaccess file on your hosting provider needs an adjustment (using CPanel, Filemanager Edit), or it won’t pass the redirect through for CSS, JS et all which are referenced through a url that includes the subdomains name, which otherwise is seen as a CORS violation

        • The .htaccess at site root does not need to change.

That’s it! You have a SETI type local hybrid cloud which you can use to offload cloud processing, provide less latency or do private LLM modelling with CPUs and GPU, and so forth.

Lots of moving pieces. It took me weeks to work through it. Support teams had pieces but not the end to end knitting of the pieces, and not the low level bindings that needed to take place to make it all happen. It required skilled knowlegeable support with lots of patience and perseverance with me as I learned the paths through the quagmire, from three vendor support teams and required escalation to their top talent:

  • Much thanks to:

    • Rogers-Shaw for the Cable Internet Modem binding to the back end Multipasss Instance.

    • Namecheap for the Subdomains with redirections, redirect SSL sets, and Multipass for the instance site configurations .

    • Multipass for guidance on how to configure Multipass instances with multiple IPs, and for a great product in the making.

It’s still early days for Multipass, but it is awesome to have a local cloud with a light agile resilient footprint.

  • On Multipass maturity, it needs a way to snapshot and a way to back up. Missing both of those makes it a risky implementation because you can do config which locks you out fo your instance.

    • For this is did a snapshot hack, so I have a recovery, and not have to start from scratch.

      • On MacOS, in terminal command line, sudo, you copy the two instance files, into a backup folder.

        • /var/root/Library/Application Support/multipassd/qemu/vault/instances/[InstanceName]/

Any fededback and refinements are well received.