Overview of GPU terminology

01-06-2026

CUDA compatability:

  • https://docs.nvidia.com/deploy/cuda-compatibility/latest/
  • the driver, the CUDA version, and your framework build all have to line up
  • two commands sort most of it out: nvidia-smi shows your driver version and the maximum CUDA version it supports, while nvcc --version shows the toolkit you actually have installed

Terminology:

  • your code
  • GPU math libraies
  • CUDA toolkit (includes compiler, math libraries, header files, and runtime)
  • GPU driver
  • GPU hardware
┌─ 1. YOUR CODE + FRAMEWORK ───────────────────────────────────┐
│  PyTorch · vLLM · llama.cpp · JAX · HF Transformers           │
│  Owner: YOU (your app code) + open source / various vendors   │
│         (TensorRT-LLM here is NVIDIA's)                        │
└──────────────────────────────┬───────────────────────────────┘
                               │  decides which math ops to run
                               v
┌─ 2. GPU MATH LIBRARIES ──────────────────────────────────────┐
│  cuBLAS · cuDNN · cuFFT · NCCL · CUTLASS · FlashAttention     │
│  Owner: mostly NVIDIA (cuBLAS/cuDNN = proprietary binaries)   │
│         some open source (CUTLASS, FlashAttention, NCCL)      │
└──────────────────────────────┬───────────────────────────────┘
                               │  calls optimized kernels
                               v
┌─ 3. CUDA TOOLKIT + RUNTIME ──────────────────────────────────┐
│  nvcc · CUDA C++ · PTX · CUDA Runtime API · Driver API        │
│  Owner: NVIDIA — proprietary platform (free to use, closed)   │
└──────────────────────────────┬───────────────────────────────┘
                               │  compiled GPU code + launch calls
                               v
┌─ 4. GPU DRIVER ──────────────────────────────────────────────┐
│  NVIDIA driver (user-mode + kernel-mode) · nvidia-smi         │
│  Owner: NVIDIA — mostly proprietary                           │
│         (open kernel module exists for newer GPUs)            │
└──────────────────────────────┬───────────────────────────────┘
                               │  hardware-specific commands
                               v
┌─ 5. GPU HARDWARE ────────────────────────────────────────────┐
│  SMs · CUDA cores · tensor cores · VRAM   (e.g. H100, RTX 4090)│
│  Owner: NVIDIA designs it · TSMC builds it · you/cloud own it  │
└──────────────────────────────────────────────────────────────┘