The NVIDIA Network Operator includes an RDMA Shared Device Plug-In and OFED Driver.
The NVIDIA GPU Operator has NVIDIA GPU Monitoring, NVIDIA Container Runtime, NVIDIA Driver, and NVIDIA Kubernetes Device Plug-In.
When deployed together, they automatically enable the GPU Direct RDMA Driver.
The NVIDIA EGX Operators are part of the NVIDIA EGX stack which contain Kubernetes, Container engine and Linux Distribution. They run bare metal virtualization.
This is the second post in a two part series.
The first post described how to integrate the NVIDIA GPU and Network Operators using preinstalled drivers.
This post describes the following tasks:
- Clean up preinstalled driver integrations
- Installing the Network Operator with a custom driver container
- Installing the GPU Operator with a custom driver container
NVIDIA Driver integration
The preinstalled driver integration method is suitable for edge deployments requiring signed drivers for secure and measured boot. Use the driver container method when the edge node has an immutable operating system. Driver containers are also appropriate when not all edge nodes have accelerators.
Clean up preinstalled driver integration
First, uninstall the previous configuration and reboot to clear the preinstalled drivers.
- Delete the test pods and network attachment.
$ kubectl delete pod roce-shared-pod pod "roce-shared-pod" deleted $ kubectl delete macvlannetwork roce-shared-macvlan-network macvlannetwork.mellanox.com "roce-shared-macvlan-network" deleted
- Uninstall the Network Operator Helm chart.
$ helm delete -n network-operator network-operator release "network-operator" uninstalled
3. Uninstall MOFED to remove the preinstalled drivers and libraries.
$ rmmod nvidia_peermem $ /etc/init.d/openibd stop Unloading HCA driver: [ OK ] $ cd ~/MLNX_OFED_LINUX-5.4-1.0.3.0-rhel7.9-x86_64 $ ./uninstall.sh
4. Remove the GPU test pod.
$ kubectl delete pod cuda-vectoradd pod "cuda-vectoradd" deleted
5. Uninstall the NVIDIA Linux driver.
$ ./NVIDIA-Linux-x86_64-470.57.02.run --uninstall
6. Remove GPU Operator.
$ helm uninstall gpu-operator-1634173044
7. Reboot.
$ sudo shutdown -r now
Install the Network Operator with a custom driver container
This section describes the steps for installing the Network Operator with a custom driver container.
The driver build script executed in the container image needs access to kernel development and packages for the target kernel. In this example the kernel development packages are provided through an Apache web server.
Once the container is built, upload it to a repository the Network Operator Helm chart can access from the host.
The GPU Operator will use the same web server to build the custom GPU Operator driver container in the next section.
- Install the Apache web server and start it.
$ sudo firewall-cmd --state not running $ sudo yum install createrepo yum-utils httpd -y $ systemctl start httpd.service && systemctl enable httpd.service && systemctl status httpd.service ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2021-10-20 18:10:43 EDT; 4h 45min ago ...
- Create a mirror of the upstream CentOS 7 Base package repository. It could take ten minutes or more to download all the CentOS Base packages to the web server. Note that the custom package repository requires 500 GB free space on the /var partition.
$ cd /var/www/html $ mkdir -p repos/centos/7/x86_64/os $ reposync -p /var/www/html/repos/centos/7/x86_64/os/ --repo=base --download-metadata -m
3. Copy the Linux kernel source files into the Base packages directory on the web server. This example assumes the custom kernel was compiled as an RPM using rpmbuild.
$ cd repos/centos/7/x86_64/os $ sudo cp ~/rpmbuild/RPMS/x86_64/*.rpm .
The Network Operator requires the following files:
- kernel-headers-${KERNEL_VERSION}
- kernel-devel-${KERNEL_VERSION}
Ensure the presence of these additional files for the GPU Operator:
- gcc-${GCC_VERSION}
- elfutils-libelf.x86_64
- elfutils-libelf-devel.x86_64
$ for i in $(rpm -q kernel-headers kernel-devel elfutils-libelf elfutils-libelf-devel gcc | grep -v "not installed"); do ls $i*; done
kernel-headers-3.10.0-1160.42.2.el7.custom.x86_64.rpm
kernel-devel-3.10.0-1160.42.2.el7.custom.x86_64.rpm
elfutils-libelf-0.176-5.el7.x86_64.rpm
elfutils-libelf-devel-0.176-5.el7.x86_64.rpm
gcc-4.8.5-44.el7.x86_64.rpm
4. Browse to the web repository to make sure it is accessible via HTTP.
$ elinks http://localhost/repos/centos/7/x86_64/os --dump
Index of /repos/centos/7/x86_64/os
[1][ICO] [2]Name [3]Last modified [4]Size [5]Description
--------------------------------------------------------------------------
[6][PARENTDIR] [7]Parent Directory -
[8][DIR] [9]base/ 2021-10-21 22:55 -
[10][DIR] [11]extras/ 2021-10-02 00:29 -
--------------------------------------------------------------------------
References
Visible links
2. http://localhost/repos/centos/7/x86_64/os/?C=N;O=D
3. http://localhost/repos/centos/7/x86_64/os/?C=M;O=A
4. http://localhost/repos/centos/7/x86_64/os/?C=S;O=A
5. http://localhost/repos/centos/7/x86_64/os/?C=D;O=A
7. http://localhost/repos/centos/7/x86_64/
9. http://localhost/repos/centos/7/x86_64/os/base/
11. http://localhost/repos/centos/7/x86_64/os/extras/
5. MOFED driver container images are built from source code in the mellanox/ofed-docker repository on Github. Clone the ofed-docker repository.
$ git clone https://github.com/Mellanox/ofed-docker.git
$ cd ofed-docker/
6. Make a build directory for the custom driver container.
$ mkdir centos
$ cd centos/
7. Create a Dockerfile that installs the MOFED dependencies and source archive into a CentOS 7.9 base image. Specify the MOFED and CentOS versions.
$ sudo cat 8. Modify the RHEL entrypoint.sh script included in the ofed-docker repository to install the custom kernel source packages from the web server. Specify the path to the base/Packages directory on the web server in the _install_prerequsities() function.
In this example 10.150.168.20 is the web server IP address created earlier in the section.
$ cp ../rhel/entrypoint.sh .
$ cat entrypoint.sh
...
# Install the kernel modules header/builtin/order files and generate the kernel version string.
_install_prerequisites() {
echo "Installing dependencies"
yum -y --releasever=7 install createrepo elfutils-libelf-devel kernel-rpm-macros numactl-libs initscripts grubby linux-firmware libtool
echo "Installing Linux kernel headers..."
rpm -ivh http://10.150.168.20/repos/centos/7/x86_64/os/base/Packages/kernel-3.10.0-1160.45.1.el7.custom.x86_64.rpm
rpm -ivh http://10.150.168.20/repos/centos/7/x86_64/os/base/Packages/kernel-devel-3.10.0-1160.45.1.el7.custom.x86_64.rpm
rpm -ivh http://10.150.168.20/repos/centos/7/x86_64/os/base/Packages/kernel-headers-3.10.0-1160.45.1.el7.custom.x86_64.rpm
# Prevent depmod from giving a WARNING about missing files
touch /lib/modules/${KVER}/modules.order
touch /lib/modules/${KVER}/modules.builtin
depmod ${KVER}
...
9. The OFED driver container mounts a directory from the host file system for sharing driver files. Create the directory.
$ mkdir -p /run/mellanox/drivers10. Upload the new CentOS driver image to a registry. This example uses an NGC private registry. Login to the registry.
$ sudo yum install -y podman
$ sudo podman login nvcr.io
Username: $oauthtoken
Password: *****************************************
Login Succeeded!
11. Use Podman to build the driver container image and push it to the registry.
$ sudo podman build --no-cache --tag nvcr.io/nv-ngc5g/mofed-5.4-1.0.3.0:centos7-amd64 .12. Tag the image and push it to the registry.
$ sudo podman images nvcr.io | grep mofed
nvcr.io/nv-ngc5g/mofed-5.4-1.0.3.0 centos7-amd64 d61e555bddda 2 minutes ago 1.13 GB
13. Override the values.yaml file included in the NVIDIA Network Operator Helm chart to install the custom driver image. Specify the image name, repository, and version for the custom driver container.
$ cat 14. Install the NVIDIA Network Operator with the new values.yaml.
$ helm install -f ./roce_shared_values_driver.yaml -n network-operator --create-namespace --wait network-operator mellanox/network-operator15. View the pods deployed by the Network Operator. The MOFED pod should be in status Running. This is the custom driver container. Note that it may take several minutes to compile the drivers before starting the pod.
$ kubectl -n nvidia-network-operator-resources get pods
NAME READY STATUS RESTARTS AGE
cni-plugins-ds-zr9kf 1/1 Running 0 10m
kube-multus-ds-w57rz 1/1 Running 0 10m
mofed-centos7-ds-cbs74 1/1 Running 0 10m
rdma-shared-dp-ds-ch8m2 1/1 Running 0 2m27s
whereabouts-z947f 1/1 Running 0 10m
16. Verify that the MOFED drivers are loaded on the host.
$ lsmod | egrep '^ib|^mlx|^rdma'
rdma_ucm 27022 0
rdma_cm 65212 1 rdma_ucm
ib_ipoib 124872 0
ib_cm 53085 2 rdma_cm,ib_ipoib
ib_umad 27744 0
mlx5_ib 384793 0
mlx5_core 1360822 1 mlx5_ib
ib_uverbs 132833 2 mlx5_ib,rdma_ucm
ib_core 357959 8 rdma_cm,ib_cm,iw_cm,mlx5_ib,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib
mlx_compat 55063 11 rdma_cm,ib_cm,iw_cm,auxiliary,mlx5_ib,ib_core,ib_umad,ib_uverbs,mlx5_core,rdma_ucm,ib_ipoib
mlxfw 22321 1 mlx5_core
17. The root filesystem of the driver container should be bind mounted to the /run/mellanox/drivers directory on the host.
$ ls /run/mellanox/drivers
anaconda-post.log bin boot dev etc home host lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
Install the GPU Operator with a custom driver container
This section describes the steps for installing the GPU Operator with a custom driver container.
Like the Network Operator, the driver build script executed by the GPU Operator container needs access to development packages for the target kernel.
This example uses the same web server that delivered development packages to the Network Operator in the previous section.
Once the container is built, upload it to a repository the GPU Operator Helm chart can access from the host. Like the Network Operator example, the GPU Operator also uses the private registry on NGC.
- Build a custom driver container.
$ cd ~
$ git clone https://gitlab.com/nvidia/container-images/driver.git
$ cd driver/centos7
2. Update the CentOS Dockerfile to use driver version 470.74. Comment out unused arguments.
$ grep ARG Dockerfile
ARG BASE_URL=http://us.download.nvidia.com/XFree86/Linux-x86_64
#ARG BASE_URL=https://us.download.nvidia.com/tesla
ARG DRIVER_VERSION=470.74
ARG DRIVER_TYPE=passthrough
ARG VGPU_LICENSE_SERVER_TYPE=FNE
ARG PUBLIC_KEY=''
#ARG PUBLIC_KEY=empty
ARG PRIVATE_KEY
3. Build the GPU driver container image and push it to NGC.
$ sudo podman build --no-cache --tag nvcr.io/nv-ngc5g/driver:470.74-centos7 .4. View the GPU driver container image.
$ podman images nvcr.io | grep 470
nvcr.io/nv-ngc5g/driver 470.74-centos7 630f0f8e77f5 2 minutes ago 1.28 GB
5. Verify that the following files are available in the custom repository created for the Network Operator installation:
- elfutils-libelf.x86_64
- elfutils-libelf-devel.x86_64
- kernel-headers-${KERNEL_VERSION}
- kernel-devel-${KERNEL_VERSION}
- gcc-${GCC_VERSION}
These files are needed to compile the driver for the custom kernel image.
$ cd /var/www/html/repos/centos/7/x86_64/os/base/Packages/
$ for i in $(rpm -q kernel-headers kernel-devel elfutils-libelf elfutils-libelf-devel gcc | grep -v "not installed"); do ls $i*; done
kernel-headers-3.10.0-1160.45.1.el7.custom.x86_64.rpm
kernel-devel-3.10.0-1160.45.1.el7.custom.x86_64.rpm
elfutils-libelf-0.176-5.el7.x86_64.rpm
elfutils-libelf-devel-0.176-5.el7.x86_64.rpm
gcc-4.8.5-44.el7.x86_64.rpm
6. Unlike the Network Operator, the GPU Operator uses a custom Yum repository configuration file. Create a Yum repo file referencing the custom mirror repository.
$ cd /var/www/html/repos
$ cat 7. The GPU Operator uses a Kubernetes ConfigMap to configure the custom repository. The ConfigMap must be available in the gpu-operator-resources namespace. Create the namespace and the ConfigMap.
$ kubectl create ns gpu-operator-resources
$ kubectl create configmap repo-config -n gpu-operator-resources --from-file=/var/www/html/repos/custom-repo.repo
configmap/repo-config created
$ kubectl describe cm -n gpu-operator-resources repo-config
Name: repo-config
Namespace: gpu-operator-resources
Labels:
Annotations:
Data
====
custom-repo.repo:
----
[base]
name=CentOS Linux $releasever - Base
baseurl=http://10.150.168.20/repos/centos/$releasever/$basearch/os/base/
gpgcheck=0
enabled=1
8. Install the GPU Operator Helm chart. Specify the custom repository location, the custom driver version, and the custom driver image name and location.
$ helm install nvidia/gpu-operator --generate-name --set driver.repoConfig.configMapName=repo-config --set driver.repoConfig.destinationDir=/etc/yum.repos.d --set driver.image=driver --set driver.repository=nvcr.io/nv-ngc5g --set-string driver.version="470.74" --set toolkit.version=1.7.1-centos7 --set operator.defaultRuntime=crio9. View the deployed pods.
$ kubectl get pods -n gpu-operator-resources
NAME READY STATUS RESTARTS AGE
gpu-feature-discovery-r6kq6 1/1 Running 0 3m33s
nvidia-container-toolkit-daemonset-62pbj 1/1 Running 0 3m33s
nvidia-cuda-validator-ljd5l 0/1 Completed 0 119s
nvidia-dcgm-9nsfx 1/1 Running 0 3m33s
nvidia-dcgm-exporter-zm82v 1/1 Running 0 3m33s
nvidia-device-plugin-daemonset-bp66r 1/1 Running 0 3m33s
nvidia-device-plugin-validator-8pbmv 0/1 Completed 0 108s
nvidia-driver-daemonset-4tx24 1/1 Running 0 3m33s
nvidia-mig-manager-kvcgc 1/1 Running 0 3m32s
nvidia-operator-validator-g9xz5 1/1 Running 0 3m33s
10. Verify the driver is loaded.
$ lsmod | grep nvidia
nvidia_modeset 1195268 0
nvidia_uvm 995356 0
nvidia 35237551 114 nvidia_modeset,nvidia_uvm
drm 456166 5 ast,ttm,drm_kms_helper,nvidia
11. Run nvidia-smi from the driver daemonset pod.
Defaulted container "nvidia-driver-ctr" out of: nvidia-driver-ctr, k8s-driver-manager (init)
Thu Oct 28 02:37:50 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.74 Driver Version: 470.74 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100-PCI... On | 00000000:23:00.0 Off | 0 |
| N/A 25C P0 32W / 250W | 0MiB / 40536MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA A100-PCI... On | 00000000:E6:00.0 Off | 0 |
| N/A 27C P0 32W / 250W | 0MiB / 40536MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
The NVIDIA peer memory driver that enables GPUDirect RDMA is not built automatically.
Repeat this process to build a custom nvidia-peermem driver container.
This additional step is needed for any Linux operating system that the nvidia-peermem installer in GPU Operator does not yet support.
The future with NVIDIA Accelerators
NVIDIA accelerators help future-proof an edge AI investment against the exponential growth of sensor data. NVIDIA operators are cloud native software that streamline accelerator deployment and management on Kubernetes. The operators support popular Kubernetes platforms out of the box and can be customized to support alternative platforms.
Recently, NVIDIA announced converged accelerators that combine DPU and GPU capability onto a single PCI device. The converged accelerators are ideal for edge AI applications with demanding compute and network performance requirements. The NVIDIA operators are being enhanced to facilitate converged accelerator deployment on Kubernetes.
Both the NVIDIA GPU Operator and Network Operator are open source software projects published under the Apache 2.0 license. NVIDIA welcomes upstream participation for both projects.
Register for GTC 2021 session, Exploring Cloud-native Edge AI, to learn more about accelerating edge AI with NVIDIA GPUs and SmartNICs.