Sometimes features introduced in a CUDA Toolkit version may actually span both the toolkit and the driver. In such cases an application that relies on features introduced in a newer version of the toolkit and driver may return the following error on older drivers: cudaErrorCallRequiresNewerDriver . As mentioned earlier, admins should then upgrade the installed driver also.

Application developers can avoid running into this problem by having the application explicitly check for the availability of features. Refer to the CUDA Compatibility Developers Guide for more details.

Applications that compile device code to PTX will not work on older drivers. If the application requires PTX then admins have to upgrade the installed driver.

PTX Developers should refer to the CUDA Compatibility Developers Guide and PTX programming guide in the CUDA C++ Programming Guide for details on this limitation.

2.3. Deployment Considerations for Minor Version Compatibility

As described, applications that directly rely only on the CUDA runtime can be deployed in the following two scenarios:

In scenario 2, system admins should be aware of the aforementioned limitations and should be able to tell why an application may be failing if they run into any issues.

Minor version compatibility has another benefit that offers flexibility in the use and deployment of libraries. Applications that use libraries that support minor version compatibility can be deployed on systems with a different version of the toolkit and libraries without recompiling the application for the difference in the library version. This holds true for both older and newer versions of the libraries provided they are all from the same major release family. Note that libraries themselves have interdependencies that should be considered. For example, each cuDNN version requires a certain version of cuBLAS.

Figure 3. NVRTC supports minor version compatibility from CUDA 11.3 onwards

However, if an application is unable to leverage the minor version compatibility due to any of the aforementioned reasons, then the Forward Compatibility model can be used as an alternative even though Forward Compatibility is mainly intended for compatibility across major toolkit versions.

3.1. Forward Compatibility Support Across Major Toolkit Versions

Increasingly, data centers and enterprises may not want to update the NVIDIA GPU Driver across major release versions due to the rigorous testing and validation that happens before any system level driver installations are done.

To support such scenarios, CUDA introduced a Forward Compatibility Upgrade path in CUDA 10.0. Figure 4. Forward Compatibility Upgrade Path

Forward Compatibility is applicable only for systems with NVIDIA Data Center GPUs or select NGC Server Ready SKUs of RTX cards. It’s mainly intended to support applications built on newer CUDA Toolkits to run on systems installed with an older NVIDIA Linux GPU driver from different major release families. This new forward-compatible upgrade path requires the use of a special package called “CUDA compat package”.

3.2.1. From Network Repositories or Local Installers

The CUDA compat package is available in the local installers or the CUDA network repositories provided by NVIDIA as cuda-compat-12.1 .

Install the package on the system using the package installer.

On Ubuntu, for example:

$ sudo apt-get install -y cuda-compat-12-1

The compat package will then be installed to the versioned toolkit location typically found in the toolkit directory. For example, for 11.8 it will be found in /usr/local/cuda-12.1/.

The cuda-compat package consists of the following files:

libcuda.so.* - the CUDA Driver

libnvidia-ptxjitcompiler.so.* - the JIT (just-in-time) compiler for PTX files

These files should be kept together as the CUDA driver is dependent on the libnvidia-ptxjitcompiler.so.* of the same version.

$ sudo apt-get install -y cuda-compat-12-1
      Selecting previously unselected package cuda-compat-12-1.
      (Reading database ... 339974 files and directories currently installed.)
      Preparing to unpack .../cuda-compat-12-0_530.30-1_amd64.deb ...
      Unpacking cuda-compat-12-1 (530.30-1) ...
      Setting up cuda-compat-12-1 (530.30-1) ...
      Processing triggers for libc-bin (2.31-0ubuntu9.2) ...

$ ls -l /usr/local/cuda/compat
      total 145676
      lrwxrwxrwx 1 root root 12 Jun 3 00:45 libcuda.so -> libcuda.so.1
      lrwxrwxrwx 1 root root 17 Jun 3 00:45 libcuda.so.1 -> libcuda.so.530.30
      -rw-r--r-- 1 root root 26255520 Jun 2 20:55 libcuda.so.530.30
      lrwxrwxrwx 1 root root 25 Jun 3 00:45 libcudadebugger.so.1 -> libcudadebugger.so.525.30
      -rw-r--r-- 1 root root 10938424 May 27 20:49 libcudadebugger.so.525.30
      lrwxrwxrwx 1 root root 19 Jun 3 00:45 libnvidia-nvvm.so -> libnvidia-nvvm.so.4
      lrwxrwxrwx 1 root root 24 Jun 3 00:45 libnvidia-nvvm.so.4 -> libnvidia-nvvm.so.525.30
      -rw-r--r-- 1 root root 92017376 Jun 2 21:21 libnvidia-nvvm.so.530.30
      lrwxrwxrwx 1 root root 34 Jun 3 00:45 libnvidia-ptxjitcompiler.so.1 -> libnvidia-ptxjitcompiler.so.530.30
      -rw-r--r-- 1 root root 19951576 Jun 2 20:38 libnvidia-ptxjitcompiler.so.530.30

$ LD_LIBRARY_PATH=/usr/local/cuda/compat:$LD_LIBRARY_PATH samples/bin/x86_64/linux/release/deviceQuery
      samples/bin/x86_64/linux/release/deviceQuery Starting...
       CUDA Device Query (Runtime API) version (CUDART static linking)
       Detected 1 CUDA Capable device(s)
       Device 0: "Tesla T4"
         CUDA Driver Version / Runtime Version 12.1 / 12.1         CUDA Capability Major/Minor version number: 9.0         
         ...<snip>...
       deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.1, CUDA Runtime Version = 12.1, NumDevs = 1
       Result = PASS

3.2.2. Manually Installing from Runfile

The cuda-compat package files can also be extracted from the appropriate datacenter driver ‘runfile’ installers ( .run ) available in NVIDIA driver downloads . To do this:

3.4. Deployment Model for Forward Compatibility

There are two models of deployment for the CUDA compat package. We recommend the use of the ‘shared’ deployment mode.

Shared deployment: Allows sharing the same compat package across installed toolkits in the system. Download and extract the latest forward compatibility package with the highest toolkit version in its name. Using RPATH, or through LD_LIBRARY_PATH or through an automatic loader (for example. ld.so.conf ), point to that package. This is the most recommended choice.

$ LD_LIBRARY_PATH=/usr/local/cuda/compat:$LD_LIBRARY_PATH

$ module load cuda/11.0

There is an important consideration to the per-application deployment approach. Older forward compatibility packages are not supported on new driver versions. Therefore the module load scripts should proactively query the system for whether the compatibility package can be used before loading the files. This is especially critical if there was a full system upgrade. In the cases where the module script cannot use CUDA compatible upgrade, a fallback path to the default system’s installed CUDA driver can provide a more consistent experience and this can be achieved using RPATH.

4. Conclusion

The CUDA driver maintains backward compatibility to continue support of applications built on older toolkits. Using a compatible minor driver version, applications build on CUDA Toolkit 11 and newer are supported on any driver from within the corresponding major release. Using the CUDA Forward Compatibility package, system administrators can run applications built using a newer toolkit even when an older driver that does not satisfy the minimum required driver version is installed on the system. This forward compatibility allows the CUDA deployments in data centers and enterprises to benefit from the faster release cadence and the latest features and performance of CUDA Toolkit.

CUDA compatibility helps users by:

5. Frequently Asked Questions

This section includes some FAQs related to CUDA compatibility.

Users can upgrade the kernel mode driver within the same branch. Sometimes this may require updating the cuda-compat* package. This use-case is supported only for drivers on LLB and LTS branches of driver for select GPUs.

The CUDA compatible upgrade is meant to ease the management of large production systems for enterprise customers. 11.4 UMD (User Mode Driver) and later will extend forward compatibility support to select NGC Ready NVIDIA RTX boards. Prior to that forward compatibility will be supported only on NVIDIA Data Center cards. It’s important to note that HW support is defined by the kernel mode driver and as such, newer CUDA drivers on their own will not enable new HW support. Refer to the following table to determine which hardware is supported by your system.

PTX and application compatibility information can be found in the CUDA Compatibility Developer’s Guide CUDA C++ Programming Guide .

Drivers have always been backwards compatible with CUDA. This means that a CUDA 11.0 application will be compatible with R450 (11.0), R455 (11.1) and beyond. CUDA applications typically statically include all the libraries (for example cudart, CUDA math libraries such as cuBLAS, cuFFT) they need, so they should work on new drivers or CUDA Toolkit installations.

In other words, since CUDA is backward compatible, existing CUDA applications can continue to be used with newer CUDA versions.

The minimum driver version required is 450.80.02.

There are some issues that admins can advise the application developers to accommodate in their code. Please refer to the Best Practices Guide for further information.

Yes. CUDA minor version compatibility and CUDA forward compatible upgrade both work when using either NGC Deep Learning Framework containers or using containers that are based on the official CUDA base images . The images include the CUDA compatible upgrade libraries and the NVIDIA Container Toolkit ( nvidia-docker2 ) has logic to correctly load the required libraries.

It is possible you are either running a wrong version of the NVIDIA driver on the system or your system does not have an NVIDIA Data Center GPU.

Trademarks

_{NVIDIA and the NVIDIA logo are trademarks and/or registered trademarks of NVIDIA Corporation in the Unites States and other countries. Other company and product names may be trademarks of the respective companies with which they are associated.}

VESA DisplayPort

_{DisplayPort and DisplayPort Compliance Logo, DisplayPort Compliance Logo for Dual-mode Sources, and DisplayPort Compliance Logo for Active Cables are trademarks owned by the Video Electronics Standards Association in the United States and other countries.}

Trademarks

_{NVIDIA and the NVIDIA logo are trademarks and/or registered trademarks of NVIDIA Corporation in the Unites States and other countries. Other company and product names may be trademarks of the respective companies with which they are associated.}