Performance Degradation After Kernel Update from 6.6 to 6.8

Project Discussion Kernel
1

Hi everyone,

I am experiencing a significant performance degradation after upgrading my kernel from version 6.6 to 6.7 and would appreciate any insights or suggestions. I have reported the issue here, but I am still looking forward to getting more feedback on this issue to see if it affects others as well.

Problem Description:
I am running a simulation application where most of the threads are using real-time scheduling (SCHED_RR), and the threads of a model are using SCHED_DEADLINE. After upgrading the kernel, I noticed that the execution time of my model has increased from 4.5ms to 6ms

What I Have Done So Far:

  1. I found this bug report and reverted the commit efa7df3e3bb5da8e6abbe37727417f32a37fba47
    mentioned in the post. Unfortunately, this did not resolve the issue.
  2. I performed a git bisect and found these two commits related to scheduling (RT and deadline),
    612f769edd06a6e42f7cd72425488e68ddaeef0a
    5fe7765997b139e2d922b58359dea181efe618f9
    After reverting these two commits, the model execution time improved to around 5 ms.
  3. I revert two more commits, and the execution time is back to 4.7ms
    63ba8422f876e32ee564ea95da9a7313b13ff0a1 [deadline server]
    efa7df3e3bb5da8e6abbe37727417f32a37fba47

Questions:

  1. Has anyone else experienced similar performance degradation after upgrading to kernel 6.8?
  2. Can anyone explain why these two commits are causing the problem? I am not very familiar with the kernel code and would appreciate any insights.
  3. Are there any additional settings or configurations I need to apply when using kernel 6.8 to avoid similar issues?

Additional Details:

  • CPU: AMD Ryzen Threadripper 3970X
  • Operating System: Ubuntu 24.04

Thanks again for any insights or suggestions

2

@woshilcb Have you been able to identify root cause or possible fixes for the performance issue? We observing performance degradation with IO’s (read and write) when we upgraded from 6.5 to 6.8 kernel part of Jammy 22.04.

3

6.7 is not an Ubuntu kernel, where did you get it from ?
(only 6.8 and 6.11 are in the archive in different HWE/non-HWE variants)

4

@ogra , I built and installed kernel locally. The problem first showed after I upgraded to Ubuntu 24.04 which uses kernel 6.8

5

Check the What I Have Done So Far: section’s point 2, I reverted those two commits and the performance is better. But I am not sure how it is related to I/O operations, since those two changes about the RT and deadline scheduling, which my application is using.

6

Thank you @woshilcb for your timely reply. Our application uses SCHED_RR for few high priority threads and also RT Throttling settings to reserve some CPU time for low priority tasks.

RT Throttling settings:

kernel.sched_rt_period_us=1600000
kernel.sched_rt_runtime_us=1520000
kernel.sched_deadline_period_max_us=1520000

I will try to revert commits listed and rerun performance benchmark. I was wondering if there is a way to overcome this perf issue with out kernel rebuild.

7

You need to file a Launchpad bug if you want someone from the Ubuntu kernel team to look at it.

Best to run ubuntu-bug linux on the affected system. Note that we will ignore the bug report if you don’t run an officially supported Ubuntu kernel.

1 Like
8

@ganjihal did you manage to resolve IO performance issue? We’re seeing the similar issue after upgrade from Ubuntu 22.04 (kernel 6.5) to Ubuntu 24.04 (either kernel 6.8 or 6.11). Downgrade would be obvious choice but kernel 6.8 has TCP improvements that really help our use case.