My server has 5 IPs. IP#1 appears to be the public outbound IP for all containers.
LXD networking is setup with the standard lxdbr0 bridge & NAT.
Any idea how I could configure a container to use a specific public outbound IP address, IP#2 to IP#5?
I’m also testing SMTP mail server software that can use multiple IPs. Is there a way for a container to be configured to have more than one public outbound IP available for the mail server to select from & use?
If you have multiple public IPs maybe you’d like to have them routed to your instances rather than doing NAT? Considering that you only have a few, maybe a separated network for your publicly exposed instance would make sense.
Some good ideas there. I’m not quite sure which way is best but am open to considering different ways of setting up. Are there any pros/cons of routing public IPs directly to containers versus doing NAT?
All this stuff I’m learning & figuring out with LXD is for a new webserver which will eventually become my primary webserver. On my current/old webserver I have everything (websites & email) running on a single shared IP.
On the new server I was planning to separate most sites into their own LXD containers. Ports 80 & 443 (for each of the 5 IPs) will all forward to a HAProxy container via LXD profile proxies. Then HAProxy will route web traffic to different containers based on the website being accessed.
I was also thinking I might setup more than one email sending server as well.
But with email servers, the outbound IP reverse DNS lookup must match the hostname used by the email server. That means with 5 IPs I can have up to 5 distinct email server hostnames to send mail from, but currently can only use 1 (the default outbound IP#1) until I figure out how to set this part up in LXD.
I don’t specifically need the public IPs inbound routed directly to the containers, but I do need the email server containers to have specific outbound IPs (so they match the email server hostname correctly during rDNS lookups).
With the standard setup using lxdbr0 bridge + NAT, is there a way to assign containers to use a specific public IP for outbound?
Re HAProxy, that’s a good way to preserve public IPs and also have another place to do fine grain access control (ACLs). That’s essentially what I do @home where I don’t have the luxury of having multiple public IPs 
FYI, HAProxy and Postfix (a MTA I’d recommend) both support the “proxy protocol” meaning that even your SMTP traffic could be proxied in a smart way.
AFAIK that’s not possible. However, if you need more control I would suggest setting ipv4.firewall=false (same for IPv6) on the network and then tweak the firewall rules on the host to your requirements. LXD injects very few rules that you can easily adapt and incorporate in your host managed ruleset.
I believe I found a solution, or at least partial solution for what I am looking to do with outbound IPs.
Was digging through the docs and found the following network bridge config option: ipv4.nat.address
When I set that value on a network bridge the public outbound traffic uses whatever public IP address I set for ipv4.nat.address
I verified this using curl ifconfig.io and also viewing that website using the links terminal browser. That website picks up the public IP used & the matching rDNS hostname. Soon as I change the ipv4.nat.address setting the new IP is immediately detected by that site.
Also verfied by sending out test emails. Using the default profile + lxdbr0, the LXD container email server sends out using the primary server IP#1. Once I set ipv4.nat.address to any of the other IPs the email server starts using the other IP in that setting for outbound SMTP.
This alone is very good and at least solves being able to run a mail server in a LXD container using any of my available IPs on the server as outbound for the rDNS checks.
Not sure if I’ll be able to get a multi outbound IP selection available for containers, but have some ideas I’m going to try out. Will post an update if I get that part figured out.
The ability to specify a per-instance NIC egress SNAT rule has been a long standing feature request, see
Oh, I didn’t know about ipv4.nat.address, thanks Tom!
I wasn’t sure ipv4.nat.address would do what I wanted, but figured I would test with one of my public IPs as the value and sure enough I ended up getting lucky!
Being specified on the bridge works fine for me, just as long as the outbound egress SNAT IP shows as the IP that I need it to for mail server rDNS checks.
The multi outbound IP selection that I was also looking into, I’m not sure I’ll be able to get that working.
My specific use case for that is using IP Pools with the Postal email server ( IP Pools - Postal - the open source mail delivery platform ). Basically what that would do for me is enable me to run a single instance of the Postal email server rather than several instances, one for each IP I need to use for a mail server.
With 5 public IPs on my server, one thing I tried was to setup 5 profiles with each profile using a different bridge. Each bridge had it’s own 10.0.#.1/24 network and each bridge also had one of my public IPs specified for ipv4.nat.address.
Then, with each profile individually added to a container, I get the egress SNAT IP for the associated bridge just fine.
I tried adding all 5 profiles to a container, but seems I could only get access to the 1 primary server IP that way.
Also tried adding 5 NICs to a single profile with each NIC using a different bridge so that each of the 5 public IPs are covered. That, still, resulted in just the main server IP being used (maybe because I added that one first, not sure).
However, I did manage to end up with 5 different (pingable) private ipv6 addresses on a container. Not quite what I was aiming for, but maybe a step closer.
+------------------------+---------+----------------------+-------------------------------------------------+-----------------+-----------+
| z-testing | RUNNING | 10.0.98.230 (eth98) | fd42:d903:28f:9fce:216:3eff:fe28:809d (eth101) | CONTAINER | 0 |
| | | | fd42:66c7:a1b4:f2ef:216:3eff:fede:47e1 (eth102) | | |
| | | | fd42:6352:981b:2d6c:216:3eff:feaa:4bc7 (eth100) | | |
| | | | fd42:288a:2300:53d4:216:3eff:fe43:8e6f (eth99) | | |
| | | | fd42:1748:8760:4c4e:216:3eff:fe28:21a6 (eth98) | | |
+------------------------+---------+----------------------+-------------------------------------------------+-----------------+-----------+
I’m not an expert in networking, so maybe it was pointless to try these things, maybe no chance for it to work. But sometimes it does work out as with ipv4.nat.address
If anyone has any other ideas on how to try doing this please let me know. Specifically, looking to be able to use all of my public server IPs as usable egress SNAT outbound IPs for the Postal mail server (with IP Pools): IP Pools - Postal - the open source mail delivery platform
If you’re looking at passing whole public IPs into instances, you could also take a look at the routed NIC type:
https://documentation.ubuntu.com/lxd/en/latest/reference/devices_nic/#nic-routed