24.04 considerably slower than 20.04 or 22.04 for some high system percentage usage cases

dsmythies · January 30, 2025, 9:10pm

Hi @simonhf , Sorry for your troubles. I do not know why if you specify “noautogroup” on the grub command line, your system is booting with “/proc/sys/kernel/sched_autogroup_enabled” set to 1. That makes no sense to me. I have been using a 6.13 mainline kernel that I compiled myself. I’ll try the default kernel.

EDIT: default kernel seems fine:

doug@s19:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.0-52-generic root=UUID=823b5f2c-38f5-4b05-92f6-6cb77b157c8a ro ipv6.disable=1 consoleblank=314 intel_pstate=active intel_pstate=no_hwp cgroup_disable=cpu noautogroup msr.allow_writes=on cpuidle.governor=teo
doug@s19:~$ uname -a
Linux s19 6.8.0-52-generic #53-Ubuntu SMP PREEMPT_DYNAMIC Sat Jan 11 00:06:25 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
doug@s19:~$ cat /proc/sys/kernel/sched_autogroup_enabled
0

dsmythies · January 30, 2025, 9:32pm

Hi @simonhf : After editing “/etc/default/grub” you ran

sudo update-grub

right?
check the actual kernel command line that the system booted with via:

doug@s19:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.0-52-generic root=UUID=823b5f2c-38f5-4b05-92f6-6cb77b157c8a ro ipv6.disable=1 consoleblank=314 intel_pstate=active intel_pstate=no_hwp cgroup_disable=cpu noautogroup msr.allow_writes=on cpuidle.governor=teo

dsmythies · January 31, 2025, 4:52pm

HI @simonhf : For what its worth: It turns out that there is a redis benchmark in the phoronix test suite of stuff, which I keep a copy of the git repository on my test server. So I tried it:

Test	Connections	6.13-default	6.13-nocpu-noauto	
get	50	2976234.20	2868160.40	-3.63%
set	50	2396052.45	2234853.77	-6.73%
get	500	2978210.90	2904632.90	-2.47%
lpop	50	3600308.00	3676051.05	2.10%
sadd	50	2815383.55	2829918.85	0.52%
set	500	2298790.67	2290305.82	-0.37%
get	1000	2880541.30	3093803.45	7.40%
lpop	500	3387303.45	3421305.75	1.00%
lpush	50	2043177.37	1989323.52	-2.64%
sadd	500	2814934.60	2776056.60	-1.38%
set	1000	2281870.12	2315653.02	1.48%
lpop	1000	2164327.90	2236368.20	3.33%
lpush	500	2075699.50	2088801.70	0.63%
sadd	1000	2645572.15	2780225.80	5.09%
lpush	1000	2195086.90	2170329.30	-1.13%

I found the test results to be very “noisy” and think that anything less than 10% is not significant.
I also found the lpop 50 and 500 test to vary wildly, also getting 2.2 million requests per second verses the 3.6 million in the table above. I don’t know why. I never saw a high percentage of system time during the test.

simonhf · January 31, 2025, 9:28pm

Hi @dsmythies ! Thanks for the info. I have also been using redis-benchmark to see the difference between Ubuntu 24.04 LTS and Ubuntu 20.04 LTS, as follows:

Commands I used on both VMs:

$ cat <<'EOF' > watch-redis-benchmark.sh ; chmod +x watch-redis-benchmark.sh
redis-benchmark -h localhost -p 6379 -c $CONNECTIONS -a $a -q -t get -r 100000 -n 1000000 & timeout 17 top -d 1 -b | perl -lane 'while(m~ ([\d\.]+) us,\s+([\d\.]+) sy,\s+([\d\.]+) ni,\s+([\d\.]+) id,\s+([\d\.]+) wa,\s+([\d\.]+) hi,\s+([\d\.]+) si,\s+([\d\.]+) st~g){ ($us, $sy, $ni, $id, $wa, $hi, $si, $st) = ($1, $2, $3, $4, $5, $6, $7, $8); $tus += $us; $tsy += $sy; $tni += $ni; $tid += $id; $twa += $wa; $thi += $hi; $tsi += $si; $tst += $st; $|++; printf qq[%u seconds: %s # %.1f us, %.1f sy, %.1f ni, %.1f id, %.1f wa, %.1f hi, %.1f si,  %.1f st\n], (++ $secs), $_, $tus, $tsy, $tni, $tid, $twa, $thi, $tsi, $tst; }' | egrep '13 seconds'; wait
EOF

On Ubuntu 24:

$ export CONNECTIONS=50; ./watch-redis-benchmark.sh;  ./watch-redis-benchmark.sh;  ./watch-redis-benchmark.sh; export CONNECTIONS=500; ./watch-redis-benchmark.sh;  ./watch-redis-benchmark.sh;  ./watch-redis-benchmark.sh; export CONNECTIONS=1000; ./watch-redis-benchmark.sh;  ./watch-redis-benchmark.sh;  ./watch-redis-benchmark.sh;
13 seconds: %Cpu(s): 27.5 us, 41.5 sy,  0.0 ni, 16.1 id,  0.0 wa,  0.0 hi, 15.0 si,  0.0 st  # 370.5 us, 544.4 sy, 1.0 ni, 185.5 id, 0.0 wa, 0.0 hi, 198.3 si,  0.0 st
GET: 65227.32 requests per second, p50=0.367 msec                   

13 seconds: %Cpu(s): 28.1 us, 41.3 sy,  0.0 ni, 13.8 id,  0.0 wa,  0.0 hi, 16.8 si,  0.0 st  # 371.3 us, 550.4 sy, 0.5 ni, 172.3 id, 0.0 wa, 0.0 hi, 205.4 si,  0.0 st
GET: 64641.24 requests per second, p50=0.375 msec                   

13 seconds: %Cpu(s): 29.0 us, 44.0 sy,  0.0 ni, 13.5 id,  0.0 wa,  0.0 hi, 13.5 si,  0.0 st  # 375.2 us, 558.7 sy, 1.0 ni, 172.5 id, 0.5 wa, 0.0 hi, 191.9 si,  0.0 st
GET: 65867.48 requests per second, p50=0.367 msec                   

13 seconds: %Cpu(s): 26.9 us, 40.6 sy,  0.0 ni, 15.7 id,  0.0 wa,  0.0 hi, 16.8 si,  0.0 st  # 327.9 us, 493.8 sy, 1.0 ni, 304.3 id, 0.0 wa, 0.0 hi, 173.1 si,  0.0 st
GET: 54106.70 requests per second, p50=4.503 msec                   

13 seconds: %Cpu(s): 28.1 us, 40.2 sy,  0.0 ni, 17.6 id,  0.0 wa,  0.0 hi, 14.1 si,  0.0 st  # 335.3 us, 494.6 sy, 1.5 ni, 298.7 id, 0.0 wa, 0.0 hi, 169.7 si,  0.0 st
GET: 53347.56 requests per second, p50=4.567 msec                   

13 seconds: %Cpu(s): 26.9 us, 41.1 sy,  0.0 ni, 16.2 id,  0.0 wa,  0.0 hi, 15.7 si,  0.0 st  # 330.4 us, 491.4 sy, 1.5 ni, 303.1 id, 0.0 wa, 0.0 hi, 173.6 si,  0.0 st
GET: 52991.36 requests per second, p50=4.607 msec                   

13 seconds: %Cpu(s): 26.4 us, 39.8 sy,  0.5 ni, 18.9 id,  0.0 wa,  0.0 hi, 14.4 si,  0.0 st  # 312.2 us, 478.5 sy, 1.0 ni, 345.1 id, 0.0 wa, 0.0 hi, 163.1 si,  0.0 st
GET: 49925.11 requests per second, p50=9.775 msec                     

13 seconds: %Cpu(s): 28.1 us, 40.9 sy,  0.0 ni, 17.7 id,  0.0 wa,  0.0 hi, 13.3 si,  0.0 st  # 309.7 us, 480.5 sy, 1.5 ni, 348.7 id, 0.0 wa, 0.0 hi, 159.5 si,  0.0 st
GET: 48393.34 requests per second, p50=10.063 msec                    

13 seconds: %Cpu(s): 27.9 us, 42.3 sy,  0.0 ni, 16.4 id,  0.0 wa,  0.0 hi, 13.4 si,  0.0 st  # 315.1 us, 477.0 sy, 1.0 ni, 349.3 id, 0.0 wa, 0.0 hi, 157.7 si,  0.0 st
GET: 49200.49 requests per second, p50=9.983 msec

On Ubuntu 20:

$ export CONNECTIONS=50; ./watch-redis-benchmark.sh;  ./watch-redis-benchmark.sh;  ./watch-redis-benchmark.sh; export CONNECTIONS=500; ./watch-redis-benchmark.sh;  ./watch-redis-benchmark.sh;  ./watch-redis-benchmark.sh; export CONNECTIONS=1000; ./watch-redis-benchmark.sh;  ./watch-redis-benchmark.sh;  ./watch-redis-benchmark.sh;
13 seconds: %Cpu(s): 31.9 us, 43.6 sy,  0.0 ni, 11.3 id,  0.0 wa,  0.0 hi, 13.2 si,  0.0 st # 387.3 us, 574.0 sy, 0.5 ni, 151.8 id, 0.0 wa, 0.0 hi, 186.3 si,  0.0 st
GET: 72379.85 requests per second, p50=0.327 msec                   

13 seconds: %Cpu(s): 36.6 us, 40.1 sy,  0.0 ni,  9.4 id,  0.0 wa,  0.0 hi, 13.9 si,  0.0 st # 424.3 us, 547.8 sy, 1.5 ni, 143.7 id, 0.0 wa, 0.0 hi, 182.7 si,  0.0 st
GET: 72077.27 requests per second, p50=0.335 msec                   

13 seconds: %Cpu(s): 27.8 us, 48.5 sy,  0.0 ni, 10.1 id,  0.0 wa,  0.0 hi, 13.6 si,  0.0 st # 404.8 us, 577.6 sy, 1.0 ni, 142.6 id, 0.0 wa, 0.0 hi, 174.3 si,  0.0 st
GET: 73088.73 requests per second, p50=0.327 msec                   

13 seconds: %Cpu(s): 31.9 us, 38.2 sy,  0.5 ni, 13.7 id,  0.0 wa,  0.0 hi, 15.7 si,  0.0 st # 360.8 us, 489.9 sy, 1.5 ni, 277.5 id, 0.0 wa, 0.0 hi, 170.5 si,  0.0 st
GET: 59045.82 requests per second, p50=4.143 msec                   

13 seconds: %Cpu(s): 33.5 us, 38.1 sy,  0.0 ni, 15.2 id,  0.0 wa,  0.0 hi, 13.2 si,  0.0 st # 360.6 us, 492.1 sy, 1.0 ni, 270.8 id, 0.0 wa, 0.0 hi, 175.6 si,  0.0 st
GET: 58534.30 requests per second, p50=4.167 msec                   

13 seconds: %Cpu(s): 31.7 us, 39.7 sy,  0.0 ni, 13.6 id,  0.0 wa,  0.0 hi, 15.1 si,  0.0 st # 354.6 us, 501.0 sy, 0.5 ni, 268.6 id, 0.0 wa, 0.0 hi, 175.6 si,  0.0 st
GET: 58934.46 requests per second, p50=4.143 msec                   

13 seconds: %Cpu(s): 28.6 us, 41.4 sy,  0.0 ni, 15.3 id,  0.0 wa,  0.0 hi, 14.8 si,  0.0 st # 333.9 us, 478.4 sy, 0.5 ni, 326.5 id, 0.0 wa, 0.0 hi, 160.9 si,  0.0 st
GET: 55493.89 requests per second, p50=8.855 msec                    

13 seconds: %Cpu(s): 27.8 us, 42.4 sy,  0.0 ni, 16.2 id,  0.0 wa,  0.0 hi, 13.6 si,  0.0 st # 338.3 us, 486.9 sy, 2.0 ni, 315.5 id, 0.0 wa, 0.0 hi, 157.1 si,  0.0 st
GET: 54662.73 requests per second, p50=9.039 msec                    

13 seconds: %Cpu(s): 27.1 us, 42.2 sy,  0.5 ni, 15.1 id,  0.0 wa,  0.0 hi, 15.1 si,  0.0 st # 332.8 us, 479.7 sy, 1.0 ni, 326.0 id, 0.0 wa, 0.0 hi, 160.7 si,  0.0 st
GET: 53653.82 requests per second, p50=9.095 msec

Note: I’m adding up the total user and system CPU (as reported by top) and showing it after 13 seconds, as an arbitrary milestone. And I’m also showing the requests per second at the end of the benchmark as reported by redis-benchmark.

Yes, it is a bit noisy, but Ubuntu 20 is clearly consistently faster for Redis get operations, or?

simonhf · January 31, 2025, 11:39pm

Hi @dsmythies ! FYI I have been wondering why Redis might be performing worse on Ubuntu 24 and wondering if it has something to do with waking up often to serve TCP traffic in general? To test this theory I installed nginx and used ab to send traffic to it. Again, I tested in triplicate, and this time nginx appears to be a little faster on Ubuntu 24. So back to the drawing board wondering why Redis is slower on Ubuntu 24…

dsmythies · February 1, 2025, 12:26am

Hi @simonhf : I observe you are only running the “GET” test. I just am not seeing anywhere near the amount of system time you are getting. If I run the "GET option for the benchmark I always get less than 1% system time. I modified the Phoronix test-definition.xml file to be more like the benchmark command line options you use, but it made no difference.

Last year this thread kind of moved past just a 20.04 verses 24.04 differences towards now cgroup and autogroup as the variants, although the change to EEVDF was also relevant.

can you comment on my post 42 above?

simonhf · February 1, 2025, 12:43am

Thanks @dsmythies . So it looks like your grub changes make little difference – or it’s too noisy – to Redis running on Ubuntu 24.

BTW any chance that you can run this command on both Ubuntu 24 and a previous Ubuntu e.g. 20, to replicate the performance issues I am seeing? Thanks!

$ redis-benchmark -h localhost -p 6379 -a $a -q -t get -r 100000 -n 1000000

simonhf · February 1, 2025, 12:48am

Hi @dsmythies, Sorry for not yet commenting on post 42. The reason is that somebody else actually made the grub changes and I am waiting to confirm with them, and or repeat the changes. Stay tuned!

Regarding the system time: I wonder if that has anything to do with the VMware environment. Also, redis-benchmark is being run on the same VM as redis, so the system time reported is for both redis-benchmark and redis, in my case.

simonhf · February 1, 2025, 2:48am

Hi again @dsmythies ! Update: The grub stuff was messed up before. There were multiple reasons why the grub changes didn’t make it through. Some of them were todo with the VMware environment changes. Anyway, it works now and after the reboot then we see:

$ cat /proc/cmdline
<regular options> cgroup_disable=cpu noautogroup

$ cat /proc/sys/kernel/sched_autogroup_enabled
0

$ seq 3 | xargs -I{} redis-benchmark -h localhost -p 6379 -a $a -q -t get -r 100000 -n 1000000
GET: 65724.61 requests per second, p50=0.375 msec                   

GET: 65915.23 requests per second, p50=0.367 msec                   

GET: 63419.59 requests per second, p50=0.383 msec

Unfortunately this does nothing to help redis performance.

But it does substantially speed up your C program to nearly the speed of Ubuntu 20:

$ dash ./ping-pong-many-parallel 0 3000000 40
pingpong: no process found

$ ps auxwww | egrep pingpong | wc --lines
81

$ # ...
202.5998 usecs/loop. EOFs: 0 # without modified grub line: 235.8666 usecs/loop. EOFs: 0, ubuntu 20: 180.8967 usecs/loop. EOFs: 0
202.5998 usecs/loop. EOFs: 0 # without modified grub line: 235.8666 usecs/loop. EOFs: 0, ubuntu 20: 180.8967 usecs/loop. EOFs: 0
202.5995 usecs/loop. EOFs: 0 # without modified grub line: 236.1925 usecs/loop. EOFs: 0, ubuntu 20: 183.7673 usecs/loop. EOFs: 0
202.5994 usecs/loop. EOFs: 0 # without modified grub line: 236.1925 usecs/loop. EOFs: 0, ubuntu 20: 183.7673 usecs/loop. EOFs: 0
202.9771 usecs/loop. EOFs: 0 # without modified grub line: 236.5349 usecs/loop. EOFs: 0, ubuntu 20: 184.5118 usecs/loop. EOFs: 0
202.9771 usecs/loop. EOFs: 0 # without modified grub line: 236.5349 usecs/loop. EOFs: 0, ubuntu 20: 184.5118 usecs/loop. EOFs: 0
203.5067 usecs/loop. EOFs: 0 # without modified grub line: 236.7970 usecs/loop. EOFs: 0, ubuntu 20: 185.1683 usecs/loop. EOFs: 0
203.5067 usecs/loop. EOFs: 0 # without modified grub line: 236.7970 usecs/loop. EOFs: 0, ubuntu 20: 185.1683 usecs/loop. EOFs: 0
203.7098 usecs/loop. EOFs: 0 # without modified grub line: 237.0187 usecs/loop. EOFs: 0, ubuntu 20: 185.3034 usecs/loop. EOFs: 0
203.7098 usecs/loop. EOFs: 0 # without modified grub line: 237.0187 usecs/loop. EOFs: 0, ubuntu 20: 185.3034 usecs/loop. EOFs: 0
203.8886 usecs/loop. EOFs: 0 # without modified grub line: 237.0415 usecs/loop. EOFs: 0, ubuntu 20: 185.5293 usecs/loop. EOFs: 0
203.8886 usecs/loop. EOFs: 0 # without modified grub line: 237.0416 usecs/loop. EOFs: 0, ubuntu 20: 185.5293 usecs/loop. EOFs: 0
204.1504 usecs/loop. EOFs: 0 # without modified grub line: 237.0620 usecs/loop. EOFs: 0, ubuntu 20: 185.6273 usecs/loop. EOFs: 0
204.1504 usecs/loop. EOFs: 0 # without modified grub line: 237.0620 usecs/loop. EOFs: 0, ubuntu 20: 185.6273 usecs/loop. EOFs: 0
204.2941 usecs/loop. EOFs: 0 # without modified grub line: 237.1598 usecs/loop. EOFs: 0, ubuntu 20: 185.6292 usecs/loop. EOFs: 0
204.2941 usecs/loop. EOFs: 0 # without modified grub line: 237.1598 usecs/loop. EOFs: 0, ubuntu 20: 185.6292 usecs/loop. EOFs: 0
204.3226 usecs/loop. EOFs: 0 # without modified grub line: 237.3042 usecs/loop. EOFs: 0, ubuntu 20: 185.6974 usecs/loop. EOFs: 0
204.3226 usecs/loop. EOFs: 0 # without modified grub line: 237.3042 usecs/loop. EOFs: 0, ubuntu 20: 185.6974 usecs/loop. EOFs: 0
204.4237 usecs/loop. EOFs: 0 # without modified grub line: 237.3535 usecs/loop. EOFs: 0, ubuntu 20: 186.1192 usecs/loop. EOFs: 0
204.4238 usecs/loop. EOFs: 0 # without modified grub line: 237.3535 usecs/loop. EOFs: 0, ubuntu 20: 186.1192 usecs/loop. EOFs: 0
204.5374 usecs/loop. EOFs: 0 # without modified grub line: 237.6999 usecs/loop. EOFs: 0, ubuntu 20: 186.1592 usecs/loop. EOFs: 0
204.5374 usecs/loop. EOFs: 0 # without modified grub line: 237.6999 usecs/loop. EOFs: 0, ubuntu 20: 186.1592 usecs/loop. EOFs: 0
204.6418 usecs/loop. EOFs: 0 # without modified grub line: 237.7379 usecs/loop. EOFs: 0, ubuntu 20: 186.8901 usecs/loop. EOFs: 0
204.6418 usecs/loop. EOFs: 0 # without modified grub line: 237.7379 usecs/loop. EOFs: 0, ubuntu 20: 186.8901 usecs/loop. EOFs: 0
204.7126 usecs/loop. EOFs: 0 # without modified grub line: 237.8664 usecs/loop. EOFs: 0, ubuntu 20: 187.0114 usecs/loop. EOFs: 0
204.7126 usecs/loop. EOFs: 0 # without modified grub line: 237.8664 usecs/loop. EOFs: 0, ubuntu 20: 187.0114 usecs/loop. EOFs: 0
204.9317 usecs/loop. EOFs: 0 # without modified grub line: 237.9694 usecs/loop. EOFs: 0, ubuntu 20: 187.1000 usecs/loop. EOFs: 0
204.9317 usecs/loop. EOFs: 0 # without modified grub line: 237.9694 usecs/loop. EOFs: 0, ubuntu 20: 187.1000 usecs/loop. EOFs: 0
205.0791 usecs/loop. EOFs: 0 # without modified grub line: 237.9740 usecs/loop. EOFs: 0, ubuntu 20: 187.1819 usecs/loop. EOFs: 0
205.0791 usecs/loop. EOFs: 0 # without modified grub line: 237.9740 usecs/loop. EOFs: 0, ubuntu 20: 187.1819 usecs/loop. EOFs: 0
205.0788 usecs/loop. EOFs: 0 # without modified grub line: 237.9982 usecs/loop. EOFs: 0, ubuntu 20: 187.2021 usecs/loop. EOFs: 0
205.0788 usecs/loop. EOFs: 0 # without modified grub line: 237.9982 usecs/loop. EOFs: 0, ubuntu 20: 187.2021 usecs/loop. EOFs: 0
205.1302 usecs/loop. EOFs: 0 # without modified grub line: 238.0590 usecs/loop. EOFs: 0, ubuntu 20: 187.2894 usecs/loop. EOFs: 0
205.1302 usecs/loop. EOFs: 0 # without modified grub line: 238.0590 usecs/loop. EOFs: 0, ubuntu 20: 187.2894 usecs/loop. EOFs: 0
205.1241 usecs/loop. EOFs: 0 # without modified grub line: 238.1404 usecs/loop. EOFs: 0, ubuntu 20: 187.4584 usecs/loop. EOFs: 0
205.1241 usecs/loop. EOFs: 0 # without modified grub line: 238.1404 usecs/loop. EOFs: 0, ubuntu 20: 187.4584 usecs/loop. EOFs: 0
205.2396 usecs/loop. EOFs: 0 # without modified grub line: 238.1640 usecs/loop. EOFs: 0, ubuntu 20: 187.5962 usecs/loop. EOFs: 0
205.2396 usecs/loop. EOFs: 0 # without modified grub line: 238.1640 usecs/loop. EOFs: 0, ubuntu 20: 187.5962 usecs/loop. EOFs: 0
205.2986 usecs/loop. EOFs: 0 # without modified grub line: 238.1652 usecs/loop. EOFs: 0, ubuntu 20: 187.6452 usecs/loop. EOFs: 0
205.2986 usecs/loop. EOFs: 0 # without modified grub line: 238.1652 usecs/loop. EOFs: 0, ubuntu 20: 187.6452 usecs/loop. EOFs: 0
205.4831 usecs/loop. EOFs: 0 # without modified grub line: 238.1993 usecs/loop. EOFs: 0, ubuntu 20: 187.7241 usecs/loop. EOFs: 0
205.4831 usecs/loop. EOFs: 0 # without modified grub line: 238.1993 usecs/loop. EOFs: 0, ubuntu 20: 187.7241 usecs/loop. EOFs: 0
205.7689 usecs/loop. EOFs: 0 # without modified grub line: 238.2745 usecs/loop. EOFs: 0, ubuntu 20: 187.7334 usecs/loop. EOFs: 0
205.7689 usecs/loop. EOFs: 0 # without modified grub line: 238.2745 usecs/loop. EOFs: 0, ubuntu 20: 187.7334 usecs/loop. EOFs: 0
205.9535 usecs/loop. EOFs: 0 # without modified grub line: 238.3439 usecs/loop. EOFs: 0, ubuntu 20: 188.0371 usecs/loop. EOFs: 0
205.9535 usecs/loop. EOFs: 0 # without modified grub line: 238.3439 usecs/loop. EOFs: 0, ubuntu 20: 188.0371 usecs/loop. EOFs: 0
206.1776 usecs/loop. EOFs: 0 # without modified grub line: 238.3574 usecs/loop. EOFs: 0, ubuntu 20: 188.1058 usecs/loop. EOFs: 0
206.1777 usecs/loop. EOFs: 0 # without modified grub line: 238.3574 usecs/loop. EOFs: 0, ubuntu 20: 188.1058 usecs/loop. EOFs: 0
206.1968 usecs/loop. EOFs: 0 # without modified grub line: 238.3626 usecs/loop. EOFs: 0, ubuntu 20: 188.1049 usecs/loop. EOFs: 0
206.1968 usecs/loop. EOFs: 0 # without modified grub line: 238.3626 usecs/loop. EOFs: 0, ubuntu 20: 188.1049 usecs/loop. EOFs: 0
206.2134 usecs/loop. EOFs: 0 # without modified grub line: 238.3667 usecs/loop. EOFs: 0, ubuntu 20: 188.2025 usecs/loop. EOFs: 0
206.2134 usecs/loop. EOFs: 0 # without modified grub line: 238.3667 usecs/loop. EOFs: 0, ubuntu 20: 188.2025 usecs/loop. EOFs: 0
206.2418 usecs/loop. EOFs: 0 # without modified grub line: 238.3810 usecs/loop. EOFs: 0, ubuntu 20: 188.2028 usecs/loop. EOFs: 0
206.2418 usecs/loop. EOFs: 0 # without modified grub line: 238.3810 usecs/loop. EOFs: 0, ubuntu 20: 188.2028 usecs/loop. EOFs: 0
206.2552 usecs/loop. EOFs: 0 # without modified grub line: 238.3854 usecs/loop. EOFs: 0, ubuntu 20: 188.2147 usecs/loop. EOFs: 0
206.2552 usecs/loop. EOFs: 0 # without modified grub line: 238.3854 usecs/loop. EOFs: 0, ubuntu 20: 188.2147 usecs/loop. EOFs: 0
206.2949 usecs/loop. EOFs: 0 # without modified grub line: 238.3873 usecs/loop. EOFs: 0, ubuntu 20: 188.3740 usecs/loop. EOFs: 0
206.2949 usecs/loop. EOFs: 0 # without modified grub line: 238.3873 usecs/loop. EOFs: 0, ubuntu 20: 188.3740 usecs/loop. EOFs: 0
206.2801 usecs/loop. EOFs: 0 # without modified grub line: 238.4390 usecs/loop. EOFs: 0, ubuntu 20: 188.4025 usecs/loop. EOFs: 0
206.2800 usecs/loop. EOFs: 0 # without modified grub line: 238.4390 usecs/loop. EOFs: 0, ubuntu 20: 188.4025 usecs/loop. EOFs: 0
206.3302 usecs/loop. EOFs: 0 # without modified grub line: 238.4614 usecs/loop. EOFs: 0, ubuntu 20: 188.4218 usecs/loop. EOFs: 0
206.3302 usecs/loop. EOFs: 0 # without modified grub line: 238.4614 usecs/loop. EOFs: 0, ubuntu 20: 188.4218 usecs/loop. EOFs: 0
206.3344 usecs/loop. EOFs: 0 # without modified grub line: 238.4893 usecs/loop. EOFs: 0, ubuntu 20: 188.4294 usecs/loop. EOFs: 0
206.3344 usecs/loop. EOFs: 0 # without modified grub line: 238.4893 usecs/loop. EOFs: 0, ubuntu 20: 188.4295 usecs/loop. EOFs: 0
206.4045 usecs/loop. EOFs: 0 # without modified grub line: 238.5087 usecs/loop. EOFs: 0, ubuntu 20: 188.4289 usecs/loop. EOFs: 0
206.4045 usecs/loop. EOFs: 0 # without modified grub line: 238.5088 usecs/loop. EOFs: 0, ubuntu 20: 188.4289 usecs/loop. EOFs: 0
206.4392 usecs/loop. EOFs: 0 # without modified grub line: 238.5133 usecs/loop. EOFs: 0, ubuntu 20: 188.4957 usecs/loop. EOFs: 0
206.4393 usecs/loop. EOFs: 0 # without modified grub line: 238.5133 usecs/loop. EOFs: 0, ubuntu 20: 188.4957 usecs/loop. EOFs: 0
206.5454 usecs/loop. EOFs: 0 # without modified grub line: 238.5465 usecs/loop. EOFs: 0, ubuntu 20: 188.4957 usecs/loop. EOFs: 0
206.5454 usecs/loop. EOFs: 0 # without modified grub line: 238.5465 usecs/loop. EOFs: 0, ubuntu 20: 188.4957 usecs/loop. EOFs: 0
206.5807 usecs/loop. EOFs: 0 # without modified grub line: 238.5784 usecs/loop. EOFs: 0, ubuntu 20: 188.5274 usecs/loop. EOFs: 0
206.5806 usecs/loop. EOFs: 0 # without modified grub line: 238.5784 usecs/loop. EOFs: 0, ubuntu 20: 188.5274 usecs/loop. EOFs: 0
206.6196 usecs/loop. EOFs: 0 # without modified grub line: 238.6299 usecs/loop. EOFs: 0, ubuntu 20: 188.5463 usecs/loop. EOFs: 0
206.6196 usecs/loop. EOFs: 0 # without modified grub line: 238.6299 usecs/loop. EOFs: 0, ubuntu 20: 188.5463 usecs/loop. EOFs: 0
206.6188 usecs/loop. EOFs: 0 # without modified grub line: 238.6366 usecs/loop. EOFs: 0, ubuntu 20: 188.5841 usecs/loop. EOFs: 0
206.6188 usecs/loop. EOFs: 0 # without modified grub line: 238.6366 usecs/loop. EOFs: 0, ubuntu 20: 188.5841 usecs/loop. EOFs: 0
206.6262 usecs/loop. EOFs: 0 # without modified grub line: 238.6609 usecs/loop. EOFs: 0, ubuntu 20: 188.5886 usecs/loop. EOFs: 0
206.6262 usecs/loop. EOFs: 0 # without modified grub line: 238.6609 usecs/loop. EOFs: 0, ubuntu 20: 188.5886 usecs/loop. EOFs: 0
206.8362 usecs/loop. EOFs: 0 # without modified grub line: 238.6951 usecs/loop. EOFs: 0, ubuntu 20: 188.6140 usecs/loop. EOFs: 0
206.8362 usecs/loop. EOFs: 0 # without modified grub line: 238.6951 usecs/loop. EOFs: 0, ubuntu 20: 188.6140 usecs/loop. EOFs: 0

So good for you, but back to the drawing board for redis

dsmythies · February 1, 2025, 4:32pm

@simonhf wrote:

BTW any chance that you can run this command on both Ubuntu 24 and a previous Ubuntu e.g. 20, to replicate the performance issues I am seeing? Thanks!

$ redis-benchmark -h localhost -p 6379 -a $a -q -t get -r 100000 -n 1000000

Sorry, I no longer have the ability to boot my test system to Ubuntu 20.04.

Okay, there seems to now be enough evidence to conclude that the redis slow down issues between Ubuntu 20.04 and 24.04 are not due to the same root cause as what I was looking into a year ago. However, it remains likely that it might be the change from the CFS (Completely Fair Scheduler) to EEVDF (Earliest Eligible Virtual Deadline First) scheduler. Among others, there were a bunch of changes to EEVDF between kernel 6.11 and 6.12-rc1. However those changes also introduced some regressions. Kernel 6.13 has the known issues fixed. My recommendation is to test with kernel 6.13. By the way, what kernel versions are you using now, both with 20.04 and 24.04? The CFS to EEVDF change was in kernel 6.6.

EDIT: I tried mainline kernel 6.5 (old CFS scheduler) but did not get a significant difference in the redis “GET” results.

simonhf · February 3, 2025, 10:28pm

By the way, what kernel versions are you using now, both with 20.04 and 24.04?

Ubuntu 20:

$ uname -r
5.4.0-204-generic

Ubuntu 24:

$ uname -r
6.8.0-51-generic

simonhf · February 5, 2025, 2:12am

Update: I’ve made some more discoveries about redis performance on Ubuntu 24. Namely, it only seems to be the localhost RPS performance which is borked. If I use redis-benchmark remotely, across the network, then redis on Ubuntu 24 actually performs better than Redis on Ubuntu 20. Below are the results of running redis-benchmark 3 times with 4 network configurations:

test  redis-benchmark  redis-server  RPS1    RPS2    RPS3
====  ===============  ============  ======  ======  ======
1     Ubuntu 20        Ubuntu 20     156543  152905  151285 <-- localhost
2     Ubuntu 24        Ubuntu 24     114916  116171  116360 <-- localhost
								   
3     Ubuntu 24        Ubuntu 20     122549  123365  120845 <-- network
4     Ubuntu 20        Ubuntu 24     134553  129416  133280 <-- network

@dsmythies I wonder if there is still a link to your C program. I know it uses named pipes and not localhost. But are named pipes and localhost conceptually similar in that they both operate “in-memory”?

dsmythies · February 5, 2025, 4:01pm

Hi @simonhf

Currently, there are too many variables between your Ubuntu 20.04 and 24.04 tests. To make progress on narrowing down your version of the issue, you need to start eliminating the variables.

What I would do next, and only with my Ubuntu 24.04 version, is to compare the Redis benchmark, localhost method, over a few mainline kernels:

6.5 Reason: before scheduler change
6.6 Reason: includes scheduler change but not commit efa7df3e3bb5
6.7 Reason: includes scheduler change and includes commit efa7df3e3bb5
6.8.0-51-generic Reason: reference kernel
6.12 Reason: Includes commit d4148aeab41 which fixes efa7df3e3bb5
6.13 Reason: as already stated, has some recent issues with EEVDF fixed. (I gave you a link to the compiled kernel in an off thread email, because the mainline one didn’t compile.)

So what’s with commits efa7df3e3bb5 and d4148aeab41 you ask? From some other work I was doing, I think they might be related. It might also be a waste of time.

simonhf · February 11, 2025, 2:37am

Hi @dsmythies, thanks for the suggestions.

Unfortunately, I am not able to easily spin up different kernel versions and test them.

However, I posted the redis issues here [1] as a github redis issue, and the user “ShooterIT” confirmed the localhost RPS redis benchmark performance drop, not only between Ubuntu 20 and Ubuntu 24, but also with Ubuntu 22. So this suggests that the redis performance drop might not be completely to do with the new scheduler in Ubuntu 24, right?

[1] [BUG] Why does redis get benchmark work up to 12% slower on Ubuntu 24 than Ubuntu 20 ? · Issue #13778 · redis/redis · GitHub

comphilip · March 23, 2025, 5:02am

autogroup regression fixed in 6.14: Linux 6.14 Sees Last Minute Fix For A Two Year Old Regression Causing A 30% Performance Drop

dsmythies · March 24, 2025, 3:00pm

If I understand correctly: The patch did not make it into kernel 6.14, just released.

I’ll try the patch, but so far I haven’t been able to get it to apply, but will try harder now that it didn’t make it into the mainline tree yet.

EDIT: Yes, the patch was included, it is just that its date is from March 14th. Compile and test is in progress.

dsmythies · March 24, 2025, 8:39pm

There is no change with respect to the performance issue for the 40 pair ping-pong test with the final version of kernel 6.14.
I don’t even get any discernible difference with my fork rate limited type tests.
Maybe about a 2% improvement for that 0 sleep fork rate example form some of the emails.