24.04 Server Network issues

Support and Help
,
1

I have a server I’m using for local speedtests. It is running IPTABLES blocking everything except my network for INPUT side. The OUTPUT has the default “allow”. SSH works for short periods then kicks me out. When I try to log back in, it times out for a few minutes, then I can get back in. DNS is broke. The only way I can get out for updates is to add a dns entry in resolve.conf. I’ve tried editing the netplan conf and reloading. Resolvectl status shows the dns entries but I get the “dns is temporarily down error” It doesn’t look like a hardware issue the best I can tell. This server has iLo running and I am able to use it to ping 8.8.8.8 when I can not access the server through SSH.

network:
  ethernets:
    eno1:
      addresses:
      - x.x.x.x/x
      routes:
        - to: default
          via: x.x.x.x
          metric: 100
      nameservers:
        addresses:
          - 8.8.8.8
          - 1.1.1.1
        search: []
    eno2:
      dhcp4: true
    eno3:
      dhcp4: true
    eno4:
      addresses:
      - x.x.x.x/x
      routes:
        - to: default
          via: x.x.x.x
          metric: 200
2

When I encounter issues where an SSH connection to an Ubuntu server works for short periods before kicking you out and timing out for a few minutes before allowing you back in, it could be due to several factors. One possibility is that the server is configured to terminate idle SSH sessions after a certain period of inactivity.

That was my issue, and this worked for me and I can’t remeber if I created this file or modified it but this is what I added:
The file is>> /etc/ssh/sshd_config

ClientAliveInterval 30
ClientAliveCountMax 5

These settings ensure that the server sends a message to the client if no data has been received from the client within the specified interval (ClientAliveInterval). If the client does not respond within the count limit (ClientAliveCountMax), the connection will be terminated. Adjusting these values can help maintain the SSH session for longer periods without interruption.
Feel free to change values for you needs. :wink:

  • Regarding the DNS issue mentioned, ensure that the hostname is correctly spelled and that the DNS resolution is functioning properly. You can verify DNS resolution by pinging the hostname or checking the /etc/hosts file to confirm that the server’s IP address is correctly mapped to its hostname.

  • Consider checking the server logs for any clues about what might be causing the disconnections. Logs such as /var/log/auth.log can provide insights into authentication attempts and potential issues.

3

I continue to have this issue. I’ve wiped UBUNTU and installed 22.04 thinking something is wrong with the IP stack in 24. In the end, I have the same issue. I can login for a bit then it drops the connection and I can’t even ping the server anymore. When I use iLO to access the server, it will still ping out, however DNS will not work. I see DNS is configured when I look at the yaml config but I cannot ping out to google. I think I’ve tried everything I could find on the web but am just having no luck. Not sure if its IPTABLES itself having some issue or what. I’m at a total loss

4

DNS issues are a bit of pain, it often points towards a network configuration issue or a problem with the DNS settings on the server itself. Ensure that the DNS settings are correctly configured in /etc/network/interfaces or /etc/netplan/*.yaml, depending on the version of Ubuntu you are using.

Make sure that the iLO port is not isolated from the network in a way that prevents DNS resolution. Checking the VLAN configuration on the switch and ensuring that the iLO port is part of the correct VLAN can resolve connectivity issues as well.

Also check that iLO firmware and drivers are up to date.

And have you checked in less /var/log/auth.log for any clues?

Good Luck, I know this kind of issues is very cumbersome.

5
  1. Create the allow-list file
sudo mkdir -p /etc/firewall
sudo nano /etc/firewall/ssh_allowed.txt
# one IP or CIDR per line
10.10.0.0/24
203.0.113.42
  1. Drop the script in place (/usr/local/sbin/firewall.sh):
#!/usr/bin/env bash
set -euo pipefail

iptables -F; iptables -X; iptables -t nat -F       
iptables -P INPUT  DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

while read -r addr; do
  [[ -n "$addr" && "$addr" != \#* ]] || continue
  iptables -A INPUT -p tcp -s "$addr" --dport 22 -j ACCEPT
done < /etc/firewall/ssh_allowed.txt

iptables -A INPUT -p icmp -j ACCEPT     

sudo chmod +x /usr/local/sbin/firewall.sh
  1. Run it once from the console/iLO to be sure you don’t lock yourself out:
sudo /usr/local/sbin/firewall.sh
  1. Make it stick after a reboot with a tiny systemd unit:
sudo tee /etc/systemd/system/firewall.service >/dev/null <<'EOF'
[Unit]
Description=Load whitelist firewall
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/firewall.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl enable firewall --now

That’s all: any time you add or remove addresses in /etc/firewall/ssh_allowed.txt, just rerun the script. Only the IPs you list can open new SSH sessions; everything else is dropped. If your list ever grows huge, swap the while-loop for an ipset, but the idea stays the same.

6

Welcome to forums @thingizkhan, That looks do-able, but could you wrap all that up in code tags netplan is very picky on indents and spacing.

network:
 version: 2
 ethernets:
   eno1:
     match:
       macaddress: ",sniped>"
     dhcp4: true
     dhcp6: true
     set-name: "eno1"

Much easier to read that way. :smiley:

1 Like
7

Sorry about that I’m still getting used to the forum formatting. I’ve edited the post and wrapped the netplan snippet in code tags does it look okay now? Thank you for your feedback

8

I get that this would make adding rules easier but what I have should not be causing the connections to drop and not be able to reconnect and for DNS to break. My outbound rules have a default ACCEPT and nothing else. Even with resolvectl showing the nameservers, I have to add a entry to resolv.conf to get dns working. Is there a known capability issue with the newer Ubuntu releases and HP servers? I can take server iso and load it on a VM and use the exact same IPTABLE rules and it works flawless. The server self tests are passing but something is surely broke somewhere. Thanks for sharing your script though! that will come in handy.

9

I have added a few log rules hoping to see something there but I don’t see the traffic I’m generating (ping and ssh) making it to the deny log. Its like the traffic isn’t making it to IPTABLES at all. I wish Ubuntu would have made it a option to use Netplan for cloud servers and leave the old default method in place for local servers. I don’t know if its the cause but it certainly hasn’t made things easier.

10

I hear you. The iptables rules aren’t the culprit the two NICs in the same subnet (each with a default route) are. Replies sometimes come back on the “other” interface, and Linux’s reverse-path check quietly drops them, which makes SSH and DNS look flaky.

Quick sanity check:

sudo ip link set eno4 down

If SSH and DNS stay solid, that’s the whole story. Either keep just one NIC/gateway or bond the two so traffic has a single, predictable path. No HP-vs-Ubuntu bug here just asymmetric routing tripping rp_filter. Hope that clears it up!

11

I’ve removed the 2nd default route and only have the subnet in place (That was a fun adventure with Netplan) but the same problem continues. The part I don’t know is if Netplan actually dropped using the 2nd route. The way that cloud config works is a bit confusing.

12

You can confirm what’s actually active in the kernel right now using:

ip route show

and

ip rule show

If you see anything unexpected like two default routes or weird priority rules it’s worth flushing them and applying the config fresh:

sudo ip route flush table main
sudo netplan apply

If you’re not using DHCP on the second NIC, consider explicitly disabling routing for it in Netplan:

  eno4:
    dhcp4: no
    routes: []
    optional: true
    link-local: []

Also, you can add this to disable gateway setting:

  gateway4: null

And finally, check this to be sure cloud-init isn’t overriding anything:

sudo cloud-init status
sudo cloud-init clean

Then reboot and see if the problem persists.

Thats all I know. I wish you solve the problem.

13

Thank You, I have looked at the ip route and it does not show the 2nd default route. The way you show the netplan config with no routes is how I removed it once I got the cloud config disabled so it would quit overwriting my custom config. Netplan states that they no longer use gateway that has to be configured some other way now. I didn’t spend much time on it once I got the default route removed. I’ve considered using a older Ubuntu without Netplan but that really isn’t going to help me. Thanks again for all your suggestions!

1 Like
14

Did you allow DNS through your IPtables rules. TCP/53 will need to be allowed in both directions