Managing NVIDIA with Dual Boot Generic Kernel and RT Kernel?

Hello,

I have a dual boot Ubuntu 24.04 setup with a 6.8.0-51-generic kernel and a 6.8.1-1014-realtime kernel at the moment. I use the standard Ubuntu “recommended driver” “Using NVIDIA driver metapackage from nvidia-driver-550 (proprietary, tested)”. This driver installs and configures correctly when I am booted into the 6.8.0-51-generic kernel but breaks when I boot 6.8.1-1014-realtime and worse it basically breaks for both kernels once I boot into the realtime kernel at which point I have to boot back into the generic kernel and reinstall/reconfigure the driver all over again. This break occurs after applying apt-get upgrades/updates etc. generally unrelated to the nvidia packages. It appears that it tries to build the nvidia module for both kernels anytime I am trying to install anything basically.

I use the real-time kernel because I run ROS2 and interface with hardware and if I could ever figure it out I would prefer to just boot and remain in the real-time kernel but the user experience under the real-time kernel offers too many issues to be investigated such as slow mouse, slow applications, and other application features breaking such as MATLAB/Simulink i.e. losing NVIDIA HW driver under the real-time kernel forces everything into “Software Rendering Mode” and the system gets sluggish. I have held my breaths for years thinking NVIDIA “is almost there” and ready to fully support their drivers with Linux but I suspect with all this NVIDIA Jetson and AI craze they may get further away from ever fully supporting their driver for real-time? IDNK but I am kind of at a cross roads on continuous reinvestment in the cards. I may be completely misinformed here but for as long as I can remember I have never been able to get a Nvidia card to work with a real-time kernel.

System Details when booted into generic kernel for example:

Functioning driver when booted into generic kernel

user@HPZ6G4-01:/var/lib/dkms/nvidia/550.120/build$ nvidia-smi 
Thu Dec 26 15:23:16 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.120                Driver Version: 550.120        CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A4000               Off |   00000000:21:00.0  On |                  Off |
| 41%   34C    P8             14W /  140W |     657MiB /  16376MiB |      7%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A     27897      G   /usr/lib/xorg/Xorg                            229MiB |
|    0   N/A  N/A     28163      G   /usr/bin/gnome-shell                          166MiB |
|    0   N/A  N/A     29304      G   /usr/bin/gnome-control-center                  21MiB |
|    0   N/A  N/A     29400      G   /usr/lib/firefox/firefox                      227MiB |
+-----------------------------------------------------------------------------------------+

System Details when booted into realtime kernel for example:

make log generated during install (apt install) of nvidia driver

user@HPZ6G4-01:/var/lib/dkms/nvidia/550.120/build$ more make.log 
DKMS make.log for nvidia-550.120 for kernel 6.8.1-1014-realtime (x86_64)
Thu Dec 26 02:54:35 PM PST 2024
make[1]: Entering directory '/usr/src/linux-headers-6.8.1-1014-realtime'
make --no-print-directory -C /usr/src/linux-headers-6.8.1-1014-realtime \
-f /usr/src/linux-headers-6.8.1-1014-realtime/Makefile modules
warning: the compiler differs from the one used to build the kernel
  The kernel was built by: x86_64-linux-gnu-gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
  You are using:           cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
make -f ./scripts/Makefile.build obj=/var/lib/dkms/nvidia/550.120/build need-builtin=1 need-modorder=1 
The kernel you are installing for is a PREEMPT_RT kernel!

The NVIDIA driver does not support real-time kernels. If you 
are using a stock distribution kernel, please install 
a variant of this kernel that does not have the PREEMPT_RT 
patch set applied; if this is a custom kernel, please 
install a standard Linux kernel.  Then try installing the 
NVIDIA kernel module again.

*** Failed PREEMPT_RT sanity check. Bailing out! ***

make[3]: *** [/var/lib/dkms/nvidia/550.120/build/Kbuild:275: preempt_rt_sanity_check] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [/usr/src/linux-headers-6.8.1-1014-realtime/Makefile:1925: /var/lib/dkms/nvidia/550.120/build] Error 2
make[1]: *** [Makefile:240: __sub-make] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-6.8.1-1014-realtime'
make: *** [Makefile:89: modules] Error 2

nvidia-smi output when booted into realtime kernel for example:

user@HPZ6G4-01:~$ nvidia-smi 
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Any advise on a proper way to manage such a dual boot setup or advice for a better setup even?

My time with the RT kernel was very short but this worked ok on my end.

sudo IGNORE_PREEMPT_RT_PRESENCE=1 apt install <*>
# # Insert downloaded .run file

But the driver was from nVidia and not a Ubuntu Driver install.
If you go this route you also need to “blacklist-nouveau” along with:

sudo update-initramfs -u

I still found it very buggy though, and have not used it since.
Some old Notes found here: [SOLVED] Plucky with my vpn again
For the record I’m still very new to tweaking the RT Kernel.

Thank you @1fallen. My experience with “I still found it very buggy, though,…” has been identical to yours. I have tried the nvidia drivers directly and even the latest “beta” etc… but their build reporting is repeatedly clear about " The NVIDIA driver does not support real-time kernels." even with PREEMPT IGNORE flags up to version 560 of the driver from the graphics PPA. The most stable and least buggy, at least for the RTXA4000, for me has been the official ubuntu release which is 550 at the moment. I have posted on Nvidia forums in the past and the responses are generally along the lines that “it works” even though the driver build reports explicitly real time is unsupported. “It works” is a vague performance metric :slight_smile:

I would like to see how @ topin89 compiled their kernel and version?
From what I’ve seen to date is failures in the Driver no mater which version used.
IMHO just not ready for Prime Time. It’s a shame I was very hopeful.

Right now this seems to be “my” best option.

 apt list --installed linux-image*
Listing... Done
linux-image-6.11.0-1002-nvidia/noble-updates,noble-security,now 6.11.0-1002.2 amd64 [installed,automatic]
linux-image-6.8.0-49-generic/noble-updates,noble-security,now 6.8.0-49.49 amd64 [installed]
linux-image-6.8.0-51-generic/noble-updates,noble-security,now 6.8.0-51.52 amd64 [installed,automatic]
linux-image-generic/noble-updates,noble-security,now 6.8.0-51.52 amd64 [installed]
linux-image-nvidia-6.11/noble-updates,noble-security,now 6.11.0-1002.2 amd64 [installed]

With “nvidia-driver-560 nvidia-dkms-560”