Installing CUDA and cuDNN on Ubuntu 24.04: Unleashing GPU Power for Deep Learning – ITFROMZERO

Table of Contents

Why is AI training still slow despite having a powerful GPU?

You just spent over $1,500 on an RTX 4090, but when you run your Python code, the GPU fans are silent while the CPU is at 100% load? This is a classic nightmare for AI developers. It feels like driving a Ferrari but having to push it because… you haven’t filled it with gas.

The problem is that the software doesn’t know how to “talk” to the hardware. Libraries like PyTorch or TensorFlow need a bridge to offload matrix operations to CUDA cores. Without the CUDA Toolkit and cuDNN, the system defaults to the CPU. As a result, processing speed drops by 20-50 times, making training even a simple model take all day.

After many trials and dozens of OS reinstalls in the lab, I’ve refined the most reliable process. This article will help you set up your environment on Ubuntu 24.04 (Noble Numbat) in just 15 minutes.

The Core of the Matter: Why CUDA and cuDNN?

To optimize performance, we need to understand the role of each component:

NVIDIA Driver: The base layer that allows Linux to recognize the existence of the graphics card.
CUDA Toolkit: A parallel computing platform. It allows you to use Python or C++ to leverage thousands of small cores inside the GPU for heavy tasks.
cuDNN (CUDA Deep Neural Network library): The “secret weapon” containing algorithms specifically optimized for Deep Learning. Without cuDNN, training architectures like Transformers or CNNs will be extremely slow.

The most common mistake is a version mismatch. An outdated driver cannot run the latest CUDA 12.x, leading to “NVIDIA driver mismatch” errors upon startup.

Choosing a Smart Installation Method

There are three common installation methods, but not all are safe:

Using the default repository (apt install nvidia-cuda-toolkit): Convenient but poor. The versions here are often 1-2 years old, lacking proper support for the RTX 40-series.
Using .run files: This method is highly prone to driver conflicts. A simple sudo apt upgrade could lead to an immediate black screen.
Using the NVIDIA Network Repository (Recommended): This is the method I use for all servers at my company. You add NVIDIA’s official repository to your system. Updates and removals are clean and safe.

A “Clean” Installation Process on Ubuntu 24.04

Step 1: Clean Up Remnants

Remove any previous failed installations to avoid library conflicts. Don’t skip this; it will save you hours of debugging later.

sudo apt-get purge nvidia* -y
sudo apt-get autoremove -y
sudo apt-get autoclean

Step 2: Install a Stable NVIDIA Driver

Instead of guessing, let Ubuntu recommend the best driver for your hardware:

sudo ubuntu-drivers devices

Look for the line labeled recommended. For example, if it’s nvidia-driver-550, install it:

sudo apt install nvidia-driver-550
sudo reboot

After rebooting, type nvidia-smi. If the stats table appears with power consumption and memory usage, you’re 50% done.

Step 3: Install CUDA Toolkit from the Official Repo

We will fetch the installation package directly from NVIDIA’s servers to ensure maximum compatibility with Ubuntu 24.04:

# Install necessary auxiliary packages
sudo apt install build-essential dkms -y

# Add NVIDIA Keyring
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb

# Update and install CUDA 12.6
sudo apt update
sudo apt install cuda-toolkit-12-6 -y

Step 4: Install cuDNN 9 Library

NVIDIA has changed the way cuDNN is distributed. You no longer need to download a zip file and manually copy files as in older 2022 guides.

sudo apt install libcudnn9-cuda-12 -y

Step 5: Set Up Environment Variables

Many users finish the installation but still see the “command not found” error when typing nvcc -V. This is because the system path hasn’t been set.

Open the configuration file:

nano ~/.bashrc

Add these two lines to the end of the file (double-check the version, 12.6 or 12.x, depending on what you installed):

export PATH=/usr/local/cuda-12.6/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Press Ctrl + O, Enter to save, and source ~/.bashrc to apply the changes immediately.

Verify the Results

Don’t celebrate just yet; check if PyTorch actually “recognizes” the GPU. Open a terminal and run the following Python command:

python3 -c "import torch; print('CUDA check:', torch.cuda.is_available()); print('Device:', torch.cuda.get_device_name(0))"

If the result is True along with your GPU’s name, congratulations! You are ready to fine-tune Llama 3 or train Stable Diffusion without worrying about software bottlenecks.

Setting up an AI environment can be frustrating. If you encounter errors like unmet dependencies or nouveau driver conflict, leave a comment below, and I’ll help you resolve them quickly.