$ wget https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-keyring_1.0-1_all.deb
$ sudo dpkg -i cuda-keyring_1.0-1_all.deb
Alternate method: Manually install the new signing key
If you can’t install the cuda-keyring package, you can install the new signing key manually (not the recommended method).
Debian, Ubuntu, WSL
$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/3bf863cc.pub
RPM distros
On a fresh installation, Fedora, RHEL, openSUSE, or SLES as
dnf/yum/zypper
prompt you to accept new keys when the repository signing key changes. Accept the change when prompted.
Replace
$distro/$arch
in the following commands with values appropriate for your OS; for example:
CUDA applications built using older NGC base containers may contain outdated repository keys. If you build Docker containers using these images as a base and update the package manager or install additional NVIDIA packages as part of your Dockerfile, these commands may fail as they would on a non-container system. To work around this, integrate the earlier commands into the Dockerfile you use to build the container.
Existing containers in which the package manager is not used to install updates are not affected by this key rotation.
Working with the NVIDIA GPU Operator
If you are a current user of the GPU Operator on Ubuntu distributions, you may be affected by the rotation of the CUDA GPG keys, where some of the containers managed by the GPU Operator may fail to start with the following error:
Stopping NVIDIA persistence daemon... Unloading NVIDIA driver kernel modules... Unmounting NVIDIA driver rootfs... Checking NVIDIA driver packages... Updating the package cache... W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 InRelease' is no longer signed.
NVIDIA is publishing new images for the driver containers by overwriting existing image tags. You can work around this error by updating the existing
clusterPolicy
to pull new images:
$ kubectl edit clusterpolicy
set driver.imagePullPolicy=Always
This step results in the GPU Operator pulling the updated images.
New installations of the GPU Operator should be unaffected by this change and do not require any
clusterPolicy
updates. If you use the GPU Operator on RHEL or OpenShift, you are also not affected by this change.
Common issues and solutions on Debian-based distros
Here are some common errors that we’ve helped people with. If you see an error not listed here, please comment below.
Duplicate .list entries
{{E: Conflicting values set for option Signed-By regarding source
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /:
/usr/share/keyrings/cuda-archive-keyring.gpg !=
E: The list of sources could not be read.}}
Solution: If you previously used
add-apt-repository
to enable the CUDA repository, then remove the duplicate entry.
sudo sed -i '/developer\.download\.nvidia\.com\/compute\/cuda\/repos/d' /etc/apt/sources.list
Also check for and remove
cuda*.list
files under the
/etc/apt/sources.d/
directory.
{{Reading package lists...
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64
InRelease: The following signatures couldn't be verified because the public key is not available:
NO_PUBKEY A4B46996 3BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64
InRelease' is no longer signed.}}
Solution: See “Duplicate .list entries” notice to install cuda-keyring package OR one of the manual enrollment methods for the 3bf863cc public key.
{{W: An error occurred during the signature verification.
The repository is not updated and the previous index files will be used.
GPG error: https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64
Release: The following signatures couldn't be verified because the public key is not available:
NO_PUBKEY F60F4B3D 7FA2AF80}}
Solution: Remove the NVIDIA machine learning repository entry, as it is no longer updated. Newer versions of cuDNN, NCCL, and TensorRT are available in the CUDA repository.
{{Packages.gz File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: XXX.XXX.XXX.XXX 443]
Hashes of expected file:
* Filesize:481481 [weak]
* SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b
* SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]
* MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]
* Release file created at: Mon, 25 Apr 2022 23:27:19 +0000
* E: Some index files failed to download. They have been ignored, or old ones used instead.}}
Solution: Report CDN issue to NVIDIA.
Related resources
Kevin Mittman is a GNU/Linux enthusiast with a passion for automation. He is a system software engineer at NVIDIA, with a focus on the installer packaging and release process for CUDA, the NVIDIA driver, and other CUDA-X products. Before joining NVIDIA, Kevin began his career in the open source community.
