DataBloom

At GTC 2021, global technology partners together with NVIDIA showcased the ways in which they leverage the ASAP2 technology to secure and accelerate modern data center workloads.

NVIDIA’s accelerated switching and packet processing (ASAP²) technology is becoming ubiquitous to supercharging networking and security for the most demanding applications.

Modern data center networks are increasingly becoming virtualized and provisioned as a service. These software-defined networks (SDN) deliver great flexibility and control, enabling you to easily scale from the premises of one data center to multi- and hybrid-cloud environments. The longstanding conflict between implementing SDN technologies and hardware-accelerated networking, namely SR-IOV, has primarily been due to fundamental differences between the two: SDN, abstracts the underlying NIC hardware and provides a virtualized network device to a virtual machine/container pod. SDN also utilizes a considerable amount of CPU processing capacity that would otherwise be used for running business applications. SR-IOV, however, does the exact opposite, providing a direct interface between the virtual machine/container pod and the NIC hardware. It bypasses the host’s CPU and operating system and thus frees up expensive CPU resources from I/O tasks. Having said that, cloud builders tend to avoid SR-IOV as it takes away live migration and sometimes requires installing a network driver on the guest operating-system. This discord between SDN and SR-IOV accelerated networking has forced system architects and network engineers to prioritize one over the other, often at the cost of poor application performance or inflexible system design, and higher TCO.

NVIDIA ASAP² — Providing the Best of Both Worlds

NVIDIA accelerated switching and packet processing technology (ASAP²), featured in ConnectX SmartNICs and BlueField data processing units (DPUs), delivers breakthrough data center networking performance, with zero CPU utilization. At the heart of ASAP²  is the eSwitch–an ASIC-embedded switch that delivers the performance and efficiency of bare-metal server networking together with the flexibility of SDN. The beauty of the eSwitch lies in how it allows the SmartNIC/DPU to handle a large portion of the packet-processing operations in the hardware, freeing up the host’s CPU and providing higher network throughput. Nearly all traffic in and out of the server—and even between the server-hosted VMs or containers—can be processed quickly by the eSwitch.

The NVIDIA ASAP² technology stack provides a range of network acceleration capabilities that enable customers to choose between maximum performance (SR-IOV) or support for legacy environments (VirtIO), etc., all while maintaining the SDN layer. BlueField DPUs also offload and isolate the SDN control plane software on the Arm cores. This provides additional CPU savings along with better control and enhanced security for cloud operators in bare metal, virtualized, and containerized environments.

The ASAP² technology is integrated upstream in the Linux kernel and in a range of leading SDN frameworks. The following diagram illustrates how NVIDIA SmartNICs and DPUs leverage ASAP² technology to accelerate the OVS stack in hardware:

At GTC 2021, global technology partners together with NVIDIA showcased the ways in which they leverage the ASAP² technology to secure and accelerate modern data center workloads. Scroll below to find information on the featured sessions and their respective use-cases.

China Mobile Accelerates Networking For Its Public Cloud Service

In this session, NVIDIA hosts industry experts from China Mobile and Nuage Networks discuss China Mobile’s  implementation of a Nuage SDN solution powered by the NVIDIA ASAP² technology, for its public cloud service. This session highlights the ASAP² vDPA acceleration technology, including its benefits and drawbacks compared to SR-IOV, as well as performance expectations.

Speakers:
Sharko Cheng, Senior Network Architect, Cloud Networking Products Department, CMCC
Mark Iskra, Director, Nokia/Nuage Networks
Ash Bhalgat, Senior Director, Cloud, Telco & Security Market Development, NVIDIA

Watch on-demand: Turbocharge Cloud-Native Applications with Virtual Data Plane Accelerated Networking

NVIDIA Secures and Accelerates GeForce NOW Cloud Gaming Platform

At NVIDIA we always strive to put our products to work before taking them to market. GeForce NOW (GFN) is NVIDIA’s cloud gaming service that runs on powerful computing platforms across 20 data centers, servicing more than 10 million subscribers. The GFN networking team has partnered internally with the NVIDIA Networking business unit to design, implement and test an OVN Kubernetes infrastructure, accelerated with NVIDIA ASAP² technology. The result is a fully integrated cloud-native SDN platform that provides a high throughput, low latency connectivity solution with built-in security. The session dives deep into the cloud platform architecture and the ASAP² acceleration stack that are geared towards providing epic gaming experience.

Speakers:
Leonid Grossman, Director, Cloud Networking, NVIDIA
Majd Debini, Software Director, Cloud Acceleration, NVIDIA

Watch on-demand: Building GeForce Now Private Cloud Networking Infrastructure using NVIDIA DPUs

Mavenir Partners With NVIDIA to Accelerate 5G UPF Applications

With 5G wireless networks being 10 times faster and supporting 10,000 times more network traffic than 4G, 5G Core is one of the most demanding data center workloads out there. Mavenir, a leading network software provider, partnered with NVIDIA to accelerate its cloud-native 5G core solution using the ASAP² technology. The session brought together two product leaders from Mavenir and NVIDIA to discuss a number of issues, including the need for hardware acceleration to deliver on the promise of 5G, how NVIDIA ASAP² has been integrated to Mavenir’s 5G User Plane Function (UPF), and how the technology is positioned to accelerate network functions (NFs) more broadly. This session is unique in the sense that ASAP² is used not only to accelerate the SDN layer, but also to accelerate the application pipeline.

Speakers:
Manikandan Sankaran, Director, Product Management, Mavenir
Gal Haim Cohen, Director, Product Management, NVIDIA

Watch on-demand: High Performance Mavenir UPF with SmartNIC Offloads

Palo Alto Networks Collaborates With NVIDIA on 5G Security Acceleration

At the forefront of cybersecurity, Palo Alto Networks has partnered with NVIDIA to integrate the ASAP² technology with the flagship PAN-OS next-generation firewall (NGFW) for addressing the stringent performance requirements of 5G networks. What they’ve built is an innovative solution that uses the NVIDIA BlueField DPU to offload policy enforcement based on application classification. Most of the data in terms of volume and bandwidth doesn’t need inspection through the firewall and is processed in hardware. The solution is dynamic in the sense that as threats evolve, the solution adapts without changing the underlying infrastructure. Initial performance results indicate ~5X improvement. The demos at the end do a great job of showing how the solution can both scale in terms of performance and system tuning,  and offload real-world traffic in conjunction with PAN’s next-generation firewall.

Speakers:
Sree Koratala, VP Product Management Mobility Security, Palo Alto Networks
John McDowall, Senior Distinguished Engineer, Palo Alto Networks
Ash Bhalgat, Senior Director, Cloud, Telco & Security Market Development, NVIDIA

Watch on-demand: Accelerated 5G Security: DPU-Based Acceleration of Next-Generation Firewalls

VMware and NVIDIA Accelerate Hybrid Cloud Networking and Security

Digital infrastructure leader VMware has partnered with NVIDIA to bring AI to every enterprise and to deliver a new architecture for the hybrid cloud. VMware project Monterey is a large-scale effort to re-architect its prominent VMware Cloud Foundation stack to use NVIDIA BlueField DPUs. A major part of this effort includes offloading, accelerating and isolating VMware’s ESXi networking to BlueField leveraging the NVIDIA ASAP² technology. This session introduces the next-generation cloud foundation architecture and the role of NVIDIA BlueField DPUs to run VMware’s NSX network and security services. Tune in to get a first look at how BlueField DPUs can be managed using existing VMware tooling, enabling new and advanced functionalities while providing familiar user experience.

Speakers:
Sudhanshu (Suds) Jain, Director Product Management, VMware
Motti Beck, Senior Director, Enterprise Market Development, NVIDIA

Watch on-demand: How to Optimize Modern Workloads over Next-Generation Hybrid Cloud Architecture

Securing and Accelerating the Data Center with NVIDIA ASAP²

Modern data center workloads demand the performance and efficiency of bare-metal server networking with the flexibility of SDN. NVIDIA and its broad partner ecosystem are leveraging the advanced ASAP² technology featured in NVIDIA’s ConnectX SmartNICs and BlueField DPUs to secure and accelerate data center workloads from cloud to edge.

Cloud computing and AI are pushing the boundaries of scale and performance for data centers. Anticipating this shift, industry leaders such as Baidu, Palo Alto Networks, Red Hat and VMware are using NVIDIA BlueField DPUs to transform their data center platforms into higher performing, more secure, agile platforms and bring differentiated products and services to Read article >

The post NVIDIA BlueField DPUs Fuel Unprecedented Data Center Transformation appeared first on The Official NVIDIA Blog.

Robotaxis are one major step closer to becoming reality. DiDi Autonomous Driving, the self-driving technology arm of mobility technology leader Didi Chuxing, announced last month a strategic partnership with Volvo Cars on autonomous vehicles for DiDi’s self-driving test fleet. Volvo’s autonomous drive-ready XC90 cars will be the first to integrate DiDi Gemini, a new self-driving Read article >

The post DiDi Chooses NVIDIA DRIVE for New Fleet of Self-Driving Robotaxis appeared first on The Official NVIDIA Blog.

The project, which runs on an NVIDIA Jetson Nano Developer Kit, helps count completed laps of a radio controlled slope glider on a course.

Steve Chang won the Jetson Project of the Month for Dragon Eye – an electronic judging system for glider races. The project, which runs on an NVIDIA Jetson Nano Developer Kit, helps count completed laps of a radio controlled slope glider on a course.

F3F is a timed speed competition for radio-controlled gliders. The goal of each pilot is to fly the glider ten laps on a 100-meter course in the shortest possible time. The top pilots in good conditions fly around ten laps in 30 to 40 seconds. To adjudicate this event, a judge needs to count the laps when the glider breaches the ends of the course, named Base A and Base B. One can imagine the judging to be a tedious and error-prone task worth automating. Steve, an accomplished F3F pilot, did just that by building Dragon Eye.

Steve’s setup for the Dragon Eye includes a Jetson Nano, two Raspberry Pi Camera Modules v2 (for different angles of view), a cooling fan for Jetson Nano, USB WiFi dongle and a few other peripherals. He wrote the code in C/C++ using Gstreamer and OpenCV libraries. The Gstreamer library was used to set up an RTSP server to stream the captured video to Jetson Nano. To identify and track the glider in the sky, Steve used the background subtraction algorithm running on Jetson Nano (implemented using OpenCV’s BackgroundSubtractorMOG2 class). Steve used the JetsonGPIO library to trigger custom alerts (e.g. to play a chime) when the glider completes a lap.  Lastly, he built an Android mobile application to control Dragon Eye and display the results of an event.

Dragon Eye – Tracking the glider and counting the laps

Steve has been improving this project over the last several months and has plans to add a 3D print layout of the camera mount to this project. We will be on the lookout for the Dragon Eye at the next F3F competition and we hope Steve’s gliders continue to soar to new heights. For developers and users to build their own version of this system, Steve has shared the bill of materials and the source code here.

Do you have a Jetson project to share? Post it on our forum for a chance to be featured here. Every month, we’ll award one Jetson AGX Xavier Developer Kit to a project that’s a cut above the rest for its application, inventiveness and creativity.

RAPIDS cuML provides scalable, GPU-accelerated machine learning models with a Python interface based on the scikit-learn API. This guide will walk through how to easily train cuML models on multi-node, multi-GPU (MNMG) clusters managed by Google’s Kubernetes Engine (GKE) platform. We will examine a subset of the available MNMG algorithms, illustrate their use of leveraging … Continued

RAPIDS cuML provides scalable, GPU-accelerated machine learning models with a Python interface based on the scikit-learn API. This guide will walk through how to easily train cuML models on multi-node, multi-GPU (MNMG) clusters managed by Google’s Kubernetes Engine (GKE) platform. We will examine a subset of the available MNMG algorithms, illustrate their use of leveraging Dask on a large public dataset and provide a series of code samples for exploring and recording their performance.

Dask as our distributed framework

Our first task will be to bring up our Dask cluster within Kubernetes. This will provide us with the ability to run distributed algorithms in a MNMG environment, and explore some of the implications this has on how we design our workflows, do analysis, and build models.

MNMG cuML and XGBoost

Once our Dask cluster is up and running, and we’ve had a chance to load some data and get a feel for the major ideas, we’ll take a look at the machine learning models RAPIDS has available, the flavors they come in: out-of-band and in-framework, and go through the process of training some of those models and looking at their performance in our cluster.

Pre-Requisites

Before we get started, we need to have a few pieces of software and a running Kubernetes cluster. I’ll provide a quick run through for spinning one up in GKE on Google’s Cloud Platform (GCP) as well as this more detailed guide; if you’re interested in more details about GCP or Kubernetes, I encourage you to to look into the links at the end of this guide.

Software

• Install Docker
• Install Kubectl
• Install Gcloud

Local Environment

• Anaconda

Python versioning and package manager. You’ll want to create a new environment (which the rapids install command below does).

• RAPIDS 0.19 python libraries

$ conda create -n rapids-core-0.19 -c rapidsai -c nvidia -c conda-forge 
    -c defaults cuml=0.19 python=3.8 cudatoolkit=11.0

• RAPIDS 0.19 docker container

We’ll modify this to be our scheduler/worker deployment container for our Dask cluster.

$ docker pull rapidsai/rapidsai-core:0.19-cuda11.0-runtime-ubuntu18.04-py3.8

• Dask-kubernetes, Dask_labextension, gcsfs, jupyterlab, tqdm, nodejs, xgboost, and seaborn python libraries

File system interface to GCP’s cloud storage (GCS) system.

$ conda activate rapids-core-0.19
$ conda install -c conda-forge dask-kubernetes gcsfs jupyterlab nodejs
$ pip install xgboost seaborn dask_labextension tqdm

• Dask LabExtension

This adds a plugin to Jupyter that lets us see what our Dask cluster is doing in real time. Very handy.

$ jupyter labextension install dask-labextension

• Add rapids-core-0.19 to JupyterLab as a selectable kernel

$ python -m ipykernel install --user --name rapids-core-0.19 --display-name "RAPIDS-0.19"

• Pull the latest notebook code and specification files from rapids cloud-ml-examples. Relevant files are found in dask/kubernetes

$ git clone https://github.com/rapidsai/cloud-ml-examples.git

At this point, you’ve got a RAPIDS-0.19 conda environment, configured with all the libraries we need, and quality of life updates for Jupyter that will make the Dask experience more interactive.

Next up: what data are we using and where do we get it?

Data

For this guide, we’ll be using a subset of the public NYC-Taxi dataset, hosted on GCS by Anaconda. The data can be accessed directly from ‘gcs://anaconda-public-data/nyc-taxi’, and can be explored easily with the gsutil utility.

$ gsutil ls -r gs://anaconda-public-data/nyc-taxi

We’ll examine a medium-sized, 150 million record set, stored in parquet format, and the larger, ~450 million record set, for 2014, 2015, 2016 saved in CSV format. This will give us a chance to observe the substantial benefit associated with selecting the proper storage format.

Kubernetes on GKE

• Create a GCP account
• Create a GKE cluster
  • Allocate a node pool with at least three nodes: one for our Dask scheduler and two for workers.
  • Nodes should be n1-standard-4 or greater with an attached GPU that has a compute capability of 6.0+ (we’ll use T4’s in this guide).
• Configure Kubectl and install a CUDA 11.0 compatible driver daemon.
• Authenticate Docker with your GCR account

Optional: The steps below need to be completed to allow distributed inference using the Forest Inference Library, or for experimenting with the parquet converted mortgage data.

• Create a Service Account + Storage Bucket

Configuring our Dask Cluster

Before we can launch our Dask cluster we need to create our scheduler/worker container and push it to GCR and update our Dask-Kubernetes configuration files to reflect your specific Kubernetes cluster.

Cluster specific items

• Navigate to the cloud-ml-examples repo you downloaded in the ‘Local Environment’ step above.

$ cd Dask/kubernetes
$ ls
Dask_Cuml_Exploration.ipynb   Dockerfile   specs

• Build your scheduler/worker container, tag it with the gcr path corresponding to your GCP project, and push your GCR repo.

$ docker build --tag gcr.io/${YOUR_PROJECT}/${YOUR_REPO}/dask-unified:0.19 --file Dockerfile .
$ docker push gcr.io/${YOUR_PROJECT}/${YOUR_REPO}/dask-unified:0.19

• Update the two yaml files sched-spec.yaml and worker-spec.yaml found in ./spec
  • Find the image entry under the containers block and set it to your GCR image path. Next, locate the limits and requests blocks and set their cpu and memory elements based on available resources in your cluster.

For example, n1-standard-4 has 4 vcpus, and 15 GB of memory, so we might configure our container specification as follows (you can find the exact amount of allocatable resources in the GCP console by looking at the ‘Nodes’ table in your cluster details).

containers:
   - image: gcr.io/${YOUR_PROJECT}/${YOUR_REPO}/Dask-unified:0.19
   … SNIP … 
   resources:
     limits:
       cpu: “3”
       memory: 13G
       nvidia.com/gpu: 1
     requests:
       cpu: “3”
       memory: 13G

Deploy the cluster

At this point, we’re finished with all the configuration elements and can start exploring the code. I’ll reference the relevant bits here, and you can refer to the underlying notebook for additional details. To get started, bring up a jupyter lab notebook instance on your workstation and open ‘Dask_cuML_Exploration.ipynb’.

• Make sure you select the RAPIDS-0.19 kernel we installed previously.

• Run the first three cells to launch your Dask cluster. These will:
  • Create scheduler and worker pod templates from ‘sched-spec.yaml’ and ‘worker-spec.yaml’.
  • Create a cluster from the pod templates, attach a Dask client, and scale up the cluster to have two workers.

cluster = KubeCluster(pod_template=worker_pod,
                      scheduler_pod_template=sched_pod)

client = Client(cluster)

n_workers = 2
cluster.scale(n_workers)

Note: This process may take 5-10 minutes for the first run, as each worker will need to pull its container.

• During this time, it can be useful to open a separate terminal window and monitor your kubernetes activity with kubectl. This will also allow you to get the external-ip of the Dask scheduler, once it’s created and being monitoring the cluster.

$ watch kubectl get all
Every 2.0s: kubectl get all                                                                                                                                                                                                                                                  	drobison-mint: Thu Feb 11 12:21:02 2021

NAME                            	READY   STATUS	RESTARTS   AGE
pod/Dask-61d38cef-e57k2r   1/1 	Running   0      	54m
pod/Dask-61d38cef-e7gbzk   1/1 	Running   0      	54m
pod/Dask-61d38cef-ebck7r   1/1 	Running   0      	56m

NAME                      TYPE       	 CLUSTER-IP   EXTERNAL-IP
service/Dask-61d38cef-e   LoadBalancer   10.44.8.55  [YOUR EXTERNAL IP]  

• Once the cluster is finished creating, you should see something like the screen below.

Running the next few cells will create a number of helper functions to help aggregate timings and scale worker counts, create some predefined data loading mechanisms for our medium and large NYC-Taxi datasets along with some pre-processing and data clean up, and create some simple visualization functions to let us explore the results.

ETL example

Most data scientists are probably aware that the choice of file format matters, but it’s not always clear how much or what the underlying trade off is. As a quick illustration, let’s look at the time required to read in ~150 million rows from CSV vs parquet data formats.

CSV

base_path = 'gcs://anaconda-public-data/nyc-taxi/csv'

with SimpleTimer() as timer_csv:
    df_csv_2014 = dask_cudf.read_csv(f'{base_path}/2014/yellow_*.csv', chunksize=25e6)
    df_csv_2014 = clean(df_csv_2014, remap, must_haves)
    df_csv_2014 = df_csv_2014.query(' and '.join(query_frags))
    
    with Dask.annotate(workers=set(workers)):
        df_csv_2014 = client.persist(collections=df_csv_2014)
        
    wait(df_csv_2014)

print(df_csv_2014.columns)
rows_csv = df_csv_2014.iloc[:,0].shape[0].compute()
print(f"CSV load took {timer_csv.elapsed/1e9} sec. For {rows_csv} rows of data => {rows_csv/(timer_csv.elapsed/1e9)} rows/sec")

On an eight GPU cluster this takes around 350 seconds, for ~155,500,000 rows.

Parquet

with SimpleTimer() as timer_parquet:
    df_parquet = dask_cudf.read_parquet(f'gs://anaconda-public-data/nyc-taxi/nyc.parquet', chunksize=25e6)
    df_parquet = clean(df_parquet, remap, must_haves)
    df_parquet = df_parquet.query(' and '.join(query_frags))
    
    with Dask.annotate(workers=set(workers)):
        df_parquet = client.persist(collections=df_parquet)
    
    wait(df_parquet)

print(df_parquet.columns)
rows_parquet = df_parquet.iloc[:,0].shape[0].compute()
print(f"Parquet load took {timer_parquet.elapsed/1e9} sec. For {rows_parquet} rows of data => {rows_parquet/(timer_parquet.elapsed/1e9)} rows/sec")

On the same eight GPU cluster, the parquet read takes around 98 seconds, for ~138,300,000 rows. A speedup of more than 3x over the CSV reads in terms of rows per second; for larger datasets this can result in a tremendous amount of time saved.

Multi-Node cuML training

Here, we’ll examine the process of training a Random Forest Regressor model across a set of workers in your cluster, examine the performance, and outline how we can scale up to more workers when necessary.

The performance sweep code goes through a fairly straightforward process.

• Calls the data loader, which reads and load-balances our dataset across Dask workers.
• Calls model.fit, ‘sample’ times, in either an X ~ y format for supervised models like RF, or just using X (KMeans, NN, etc..), and records the resulting timings.
• Calls model. predict, ‘sample’ times, for all rows in X, and records the resulting timings.

Two-Node performance

From the RAPIDS’ documentation: This distributed algorithm uses an embarrassingly-parallel approach. For a forest with N trees being built on w workers, each worker simply builds N/w trees on the data it has available locally. In many cases, partitioning the data so that each worker builds trees on a subset of the total dataset works well, but it generally requires the data to be well-shuffled in advance. Alternatively, callers can replicate all of the data across workers so that rf.fit receives w partitions, each containing the same data. This would produce results approximately identical to single-GPU fitting.

from cuml.dask.ensemble import RandomForestRegressor

rf_kwargs = {
    "workers": client.has_what().keys(),
    "n_estimators": 10,
    "max_depth": 6
}
rf_csv_path = f"./{out_prefix}_random_forest_classifier.csv"

benchmark_sweep(client=client, model=RandomForestRegressor,
                **benchmark_kwargs,
                out_path=rf_csv_path,
                response_dtype=np.int32,
                model_kwargs=rf_kwargs)

visualize_csv_data(rf_csv_path)

Running this, you’ll see the following:

Starting weak-scaling performance sweep for:
 model      : 
 data loader: .
Configuration
==========================
Worker counts             : [2]
Fit/Predict samples       : 5
Data load samples         : 1
- Max data fraction       : 1.00
 - Train                  : 1.00
 - Infer                  : 1.00
Model fit                 : X ~ y
- Response DType          : 
Writing results to        : ./taxi_medium_random_forest_regression.csv
- Method                  : append


Sampling  load times with 2 workers.  With 12.5 percent of total data

100%|██████████| 1/1 [17:19, samples, to  workers with a mean time of 1039.3022 sec.
Sweeping  'fit' with  workers. Sampling   times with 12.5 percent of total data.

100%|██████████| 5/5 [06:55, 'fit' samples using   workers, with a mean time of 83.0431 sec.
Sweeping  'predict' with  workers. Sampling  times with 12.5 percent of total data.

100%|██████████| 5/5 [07:23, 'predict' samples using   workers, with a mean time of 88.6003 sec.


  hardware  n_workers     type     ci.low    ci.high
0        T4          2      fit  82.610233  83.476041
1        T4          2  predict  86.701627  90.498879

Note that if we wanted to check our algorithm performance for multiple hardware types, we could rerun the previous commands on a different cluster configuration, and since we’re set to append to our existing data set we would then see something similar to the graph below. (See the Vis and Analysis section of the notebook for more information).

Scaling up and out

At this point we’ve trained our random forest using two workers and collected some data; now let’s assume we want to scale up our workflow to support a larger dataset.

There are two possible scaling cases we need to consider, the first is that we want to scale our worker counts, and our Kubernetes cluster already has sufficient resources; in this case, all we need to do is tell our KubeCluster object to scale the cluster, and it will spin up additional worker pods and connect them to the scheduler.

n_workers = 16
cluster.scale(n_workers)

The second scenario is one where we don’t have sufficient Kubernetes resources to launch additional workers. In this case, we’ll need to go back to GKE and increase the size of our node pool before we can scale up our worker count. Once we’ve done that, we can go back and update our sweep configuration to run with four and eight workers, and kick off another run. Examining the results, we see the relatively flat profile that we would expect for a weak scaling run.

… SNIP … 
  hardware  n_workers     type     ci.low    ci.high
0        T4          2      fit  82.610233  83.476041
2        T4          8  predict  21.372635  25.744045
3        T4         16      fit   3.417929   3.491950
6        T4         16  predict  17.952964  21.310312
8        T4          4  predict  45.985099  47.991526
9        T4          2  predict  86.701627  90.498879
10       T4          8      fit   8.470403   9.081820
11       T4          4      fit  30.791707  31.451253

Similarly, if we want to gather additional scaling data for another hardware type, say V100’s, we can rebuild our cluster, selecting V100s instead of T4s, and re-run our performance sweeps to produce the following.

… SNIP … 
  hardware  n_workers     type     ci.low    ci.high
1      A100          4  predict  36.307192  39.593791
2        T4          8  predict  21.372635  25.744045
3        T4         16      fit   3.417929   3.491950
4      A100         16      fit   1.438821   1.541184
7      A100          8  predict  23.161591  25.073527
9        T4          8      fit   8.470403   9.081820
10     A100          4      fit   5.009861   5.124773
11       T4          4      fit  30.791707  31.451253
13       T4          2      fit  82.610233  83.476041
14     A100          2      fit  13.178733  13.259799
15     A100          2  predict  42.331923  44.573160
17     A100         16  predict  17.292678  17.742979
19       T4         16  predict  17.952964  21.310312
21     A100          8      fit   2.302896   2.337939
22       T4          4  predict  45.985099  47.991526
23       T4          2  predict  86.701627  90.498879

XGBoost performance

Following similar steps, we can evaluate cluster performance for all our other algorithms, including XGBoost. The following example is trained on a subset of the much larger ‘mortgage’ dataset, which is available here. Note that because this dataset is not publicly hosted on GCP, some additional steps are required to pull the data, push to a private GCP bucket, convert the dataset to Parquet. The setup required for GCP/GKE is covered in the optional portion of the ‘Kubernetes on GKE’ section of this document; the scripts for converting the mortgage dataset to parquet can be found here.

In addition to the steps described above, we will also utilize the RAPIDS Forest Inference Library (FIL) framework for accelerated inference of our trained XGBoost model. The process for this is somewhat different from what occurs with RandomForest. After training our initial XGBoost model is fit, we will save the model to a centralized GCP bucket, and subsequently instantiate the model as a FIL object on each of our available workers. Once that step is completed, we can perform FIL based inference locally on each worker for its portion of the dataset.

Conclusion

Congratulations! At this point you’ve gone through the process of spinning up a Dask cluster in GKE, loaded a substantial dataset, performed distributed training using multiple nodes and GPUs, and built familiarity with the Dask ecosystem and monitoring tools for Jupyter.

Going forward, this should provide you with a basic template for utilizing Dask, RAPIDS, and XGboost with your own datasets to build and evaluate your workflow in Kubernetes.

For more information about the technologies we’ve used, such as RAPIDS, Dask, and Kubernetes, check out the links below.

• RAPIDS Official Website
• RAPIDS HPO Webpage
• RAPIDS Github
• RAPIDS Cloud-ML Examples
• Dask Official Website
• Kubernetes Official Website

$ sudo docker run -it tensorflow/tensorflow:latest-gpu-jupyter bash ________ _______________ ___ __/__________________________________ ____/__ /________ __ __ / _ _ _ __ _ ___/ __ _ ___/_ /_ __ /_ __ _ | /| / / _ / / __/ / / /(__ )/ /_/ / / _ __/ _ / / /_/ /_ |/ |/ / /_/ ___//_/ /_//____/ ____//_/ /_/ /_/ ____/____/|__/ WARNING: You are running this container as root, which can cause new files in mounted volumes to be created as the root user on your host machine. To avoid this, run the container by specifying your user's userid: $ docker run -u $(id -u):$(id -g) args... root@4d6368436b20:/tf# python -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))" Illegal instruction (core dumped) 

submitted by /u/notooth1
[visit reddit] [comments]

myModelCheckpoint = ModelCheckpoint("dirname") for _ in range( nIterations ): x_train, y_train = getTrainingDataFromGenerators() model.fit( x_train, y_train, ... epochs=10, callbacks=[myModelCheckpoint ]) 

submitted by /u/Simusid
[visit reddit] [comments]

The latest version provides full support for running deep learning, automotive and scientific analysis on NVIDIA’s Ampere GPUs.

Announcing the availability of MATLAB 2021a on the NGC catalog, NVIDIA’s hub of GPU-optimized AI and HPC software. The latest version provides full support for running deep learning, automotive and scientific analysis on NVIDIA’s Ampere GPUs.

In addition to supporting Ampere GPUs, the latest version also includes the following features and benefits: 

• Support for NVIDIA TensorRT 7.2.x in GPU Coder for generating optimized CUDA code
• Support for importing TensorFlow 2 models directly into MATLAB
• New Experiment Manager to design and run training experiments and compare deep learning networks
• Dynamic controls in live scripts as well as a new task for adding plots to live scripts without writing any code

Download the MATLAB R2021a container from the NGC catalog. 

With the continued growth of AI models and data sets and the rise of real-time applications, getting optimal inference performance has never been more important. In this post, you learn how to get the best natural language inference performance from AWS G4dn instance powered by NVIDIA T4 GPUs, and how to deploy BERT networks easily using NVIDIA Triton Inference Server.

As the explosive growth of AI models continues unabated, natural language processing and understanding are at the forefront of this growth. As the industry heads toward trillion-parameter models and beyond, acceleration for AI inference is now a must-have.

Many organizations deploy these services in the cloud and seek to get optimal performance and utility out of every instance they rent. Instances like the AWS G4dn, powered by NVIDIA T4 GPUs, is a great platform for delivering AI inference to cutting-edge applications. The combination of Tensor Core technology, TensorRT, INT8 precision, and NVIDIA Triton Inference Server team up to get the best inference performance from AWS.

For starters, here are the dollars and cents. Running BERT-based networks, AWS customers can get a million sentences inferenced for about a dime. Using BERT Large, which is about three times larger than BERT Base, you can get a million sentences inferenced for around 30 cents. The efficiency of the T4 GPU that powers the AWS g4dn.xlarge instance means you can cost effectively deploy smart, powerful natural language applications to attract new customers, and deliver great experiences to existing customers.

Deploying inference-powered applications is still sometimes harder than it must be. To that end, we created NVIDIA Triton Inference Server. This open-source server software eases deployment, with automatic load balancing, automatic scaling, and dynamic batching. This last feature is especially useful as many natural language AI applications must operate in real time.

Currently, NVIDIA Triton supports a variety of major AI frameworks, including TensorFlow, TensorRT, PyTorch, and ONNX. You can also implement your own custom inference workload by using the Python and C++ custom backend.

With the new feature introduced in the NVIDIA Triton tools model analyzer, you can set a latency budget of five milliseconds. NVIDIA Triton automatically sets the optimal batch size for best throughput while maintaining that latency budget. In addition, NVIDIA Triton is tightly coupled with Kubernetes. It can be used with cloud provider-managed Kubernetes services like Amazon EKS, Google Kubernetes Engine, and Azure Kubernetes Service.

BERT inference performance

Because language models are often used in real-time applications, we discuss performance at several latency targets, specifically 5ms and 10ms. To simulate a real-world application, assume that there are multiple end users all sending inference requests with a batch size of one simultaneously. What’s of interest is how many requests can be handled per second.

For these measurements on G4dn, we used NVIDIA Triton to serve the BERT QA model with a sequence length of 128 and precision of INT8. Using INT8 precision, we saw up to an 80% performance improvement compared to FP16, which translates into more simultaneous requests at any given latency requirement.

Much higher throughput can be obtained with the NVIDIA Triton optimizations of concurrent model execution and dynamic batching. You can also make use of the model analyzer tool to help you find the optimal configurations to maximize the throughput under the required latency of 5 ms and 10 ms.

For BERT Base, you can see that T4 can get nearly 1,800 sentences/sec within a 10ms latency budget. T4 very quickly achieves its maximum throughput, and so the real-time throughput is about the same as the high batch throughput. This performance means that a single T4 GPU can simultaneously deliver answers to nearly 1,800 simultaneous requests and deliver a million of these answers for less than a dime, making it a cost-effective solution.

BERT-Large is about three times larger than BERT-Base and can deliver more accurate and refined answers. With this model, T4 on G4dn can deliver 449 samples per second within the 10ms latency limit, and 525 sentences/sec for batch throughput. In terms of cost per million inferences, this translates into an instance cost of 33 cents for real-time and 28 cents for batch throughput, again delivering great performance/dollar.

Optimal inference with TensorRT

NVIDIA TensorRT plays a key role in getting the most performance and value out of AWS G4 instances. This inference SDK delivers high-performance, deep learning inference. It includes a deep learning inference optimizer and runtime that brings low latency and high throughput for deep learning inference applications. Figure 2 shows the major features:

1. Reduce mixed precision: Maximizes throughput by quantizing models to INT8 while preserving accuracy.
2. Layer and tensor fusion: Optimized use of GPU memory and bandwidth by fusing nodes in a kernel.
3. Kernel auto-tuning: Selects best layers and algorithms based on the target GPU platform.
4. Dynamic tensor memory: Minimizes memory footprint and reuses memory for tensors efficiently.
5. Multi-stream execution: Uses a scalable design to process multiple input streams in parallel.
6. Time fusion: Optimizes recurrent neural networks over time with dynamically generated kernels.

TensorRT maximizes throughput by quantizing models to INT8 while preserving accuracy, and automatically selects best data layers and algorithms that are optimized for the target GPU platform.

TensorRT and NVIDIA Triton Inference Server software are both available from NGC Catalog, the curated set of NVIDIA GPU-optimized software for AI, HPC, and visualization. The NGC Catalog consists of containers, pretrained models, Helm charts for Kubernetes deployments, and industry-specific AI toolkits with SDKs. TensorRT and NVIDIA Triton are also both available in the NGC Catalog in AWS Marketplace, making it even easier to use these resources on AWS G4 instances.

Amazon EC2 G4 instances

AWS offers the G4dn Instance based on NVIDIA T4 GPUs, and describes G4dn as “the lowest cost GPU-based instances in the cloud for machine learning inference and small scale training.”

Amazon EC2 offers a variety of G4 instances with one or multiple GPUs, and with different amounts of vCPU and memory. You can perform BERT inference below 5 ms on a single T4 GPU with 16 GB, such as on a g4dn.xlarge instance. The cost of this instance at the time of publication is $0.526 per hour on demand in the US East (N. Virginia) Region

Running BERT on AWS G4dn

Here’s how to get the most performance from the popular language model BERT, a transformer-based model introduced by Google a few years ago. We discuss performance for both BERT-Base and BERT-Large and then walk through how to set up NVIDIA Triton to perform inferences on both models.

To experience the outstanding performance shown earlier, follow the detailed steps in the TensorRT demo. To save all the efforts needed for complicated environment setup and configuring, you can start directly with updated monthly, performance-optimized containers available on NGC.

Launch an EC2 instance. Select the Deep Learning AMI (Ubuntu 18.04) version 43.0 to run on a g4dn.xlarge instance with at least 150G of storage space. Log into your instance.

Clone TensorRT repository into your local environment:

git clone -b master https://github.com/nvidia/TensorRT TensorRT 

Pull and launch the container:

docker run --gpus all -it --rm -v $HOME/TensorRT:/workspace/TensorRT nvcr.io/nvidia/tensorflow:21.04-tf1-py3 

Set up the NGC command line interface and download dataset as well as models. To set up the NGC command line interface, follow the download instructions based on your OS (AMD64 Linux, in this case).

Change to the BERT directory:

cd /workspace/TensorRT/demo/BERT 

Download SQuAD v2.0 training and dev dataset.

bash ./scripts/download_squad.sh v2_0 

Download TensorFlow checkpoints for BERT base model with sequence length 128, fine-tuned for SQuAD v2.0. It takes few minutes to download the model.

bash scripts/download_model.sh base 

Install related packages and Build the TensorRT engine. To build an engine, follow these steps. Install the required package:

pip install pycuda 

Create a directory to store the engines:

mkdir -p engines 

Run the builder.py script to build the engine with FP16 precision:

python3 builder.py 
         -m models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_128_v19.03.1/model.ckpt 
         -o engines/bert_base_128.engine 
         -b 1 -s 128 --fp16 
         -c models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_128_v19.03.1 

Run the builder.py script to build the engine with INT8 precision to obtain the best performance, as shown earlier:

python3 builder.py 
         -m models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_128_v19.03.1/model.ckpt 
         -o engines/bert_base_128_int8mix.engine 
         -b 1 -s 128 --int8 --fp16 --strict  
         -c models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_128_v19.03.1 
         --squad-json ./squad/train-v2.0.json 
         -v models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_128_v19.03.1/vocab.txt 
         --calib-num 100 -iln -imh 

Test and benchmark the TensorRT engine that you just created. For benchmarking the model generated at 5.c, run the following command:

python3 perf.py -e /workspace/TensorRT/demo/BERT/engines/bert_base_128.engine -b 1 -s 128 

For benchmarking the model generated at 5.d, run the following command:

python3 perf.py -e /workspace/TensorRT/demo/BERT/engines/bert_base_128_int8mix.engine -b 1 -s 128 

For more information about performance results on various GPU architectures and settings, see BERT Inference Using TensorRT: Results.

Deploy the BERT QA model for inference with NVIDIA Triton Inference Server

NVIDIA Triton supports the following optimization modes:

• Concurrent model execution: Enables multiple models, or multiple instances of the same model, to execute in parallel on the same GPU or on multiple GPUs to exploit the parallelism of GPU better.
• Dynamic batching: Instruct the server to wait a predefined amount of time to combine individual inference requests into a preferred batch size preconfigured to enhance GPU utilization and improve inference throughput.

For more information about framework-specific optimization, see Optimization.

In this section, we show you how to deploy the TensorRT model with NVIDIA Triton Inference Server and turn on concurrent model execution. We also demonstrate dynamic batching with only a few lines of code. Follow these steps on the g4dn.xlarge instance launched earlier.

In the first step, you regenerate the model files with a larger batch size to enable NVIDIA Triton dynamic batching optimizations. This is supposed to run in the same container as the one in previous sections.

Regenerate the TensorRT engine files with a larger batch size:

mkdir -p triton_engines
 python3 builder.py 
         -m models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_128_v19.03.1/model.ckpt 
         -o triton_engines/bert_base_128_int8mix.engine 
         -b 16 -s 128 --int8 --fp16 --strict  
         -c models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_128_v19.03.1 
         --squad-json ./squad/train-v2.0.json 
         -v models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_128_v19.03.1/vocab.txt 
         --calib-num 100 -iln -imh 

Exit the current Docker environment and create a directory for triton_serving by running the following commands:

exit
 mkdir -p $HOME/triton_serving/bert_base_qa/1
 cp $HOME/TensorRT/demo/BERT/triton_engines/bert_base_128_int8mix.engine $HOME/triton_serving/bert_base_qa/1/model.plan 

You can change the bert_base_128_int8mix.engine to other engine files that you would like to serve on NVIDIA Triton.

It should be in the following format if everything works correctly:

triton_serving
 └── bert_base_qa
     └── config.pbtxt (optional)
     └── 1
         └── model.plan 

In this format, triton_serving is the model repository containing all your models, bert_base_qa is the model name, and 1 is the version number.

If you don’t know what to put into the config.pbtxt file yet, you may use the --strict-model-config False flag to let NVIDIA Triton serve the model with an automatically generated configuration.

In addition to the default configuration automatically generated by the NVIDIA Triton server, we recommend finding an optimal configuration based on the actual workload that users need. Download our example config.pbtxt file.

As you can see from the config.pbtxt file, you only need four lines of code to enable dynamic batching:

dynamic_batching {
   preferred_batch_size: 4
   max_queue_delay_microseconds: 2000
 } 

Here, the preferred_batch_size option means the preferred batch size that you would like to combine your input requests into. The max_queue_delay_microseconds option is how long the NVIDIA Triton server waits when the preferred size cannot be created from the available requests.

For concurrent model execution, directly specify the model concurrency per GPU by changing the count number in the instance_group.

instance_group {
   count: 2
   kind: KIND_GPU
 } 

For more information about the configuration files, see Model Configuration.

Start the NVIDIA Triton server by running the following command:

docker run --gpus all --rm -p8000:8000 -p8001:8001 -p8002:8002 -v $HOME/triton_serving/:/models nvcr.io/nvidia/tritonserver:21.04-py3 tritonserver --model-repository=/models (--strict-model-config False) 

The --strict-model-config False is only needed if you are not including the config.pbtxt file in your NVIDIA Triton server directory.

With that, congratulations on having your first NVIDIA Triton server running! Feel free to use the HTTP and grpc protocols to send your request or use the Performance Analyzer tool to test the server performance.

NVIDIA Triton performance benchmarking with perf_analyzer

For the following benchmark, you use the perf_analyzer application to generate concurrent inference requests and measure the throughput and latency of those requests. By default, perf_analyzer sends requests with concurrency number 1 and batch size 1. The whole process works as follows:

• The perf_analyzer sends one inference request to NVIDIA Triton, waits for the response, and only sends the subsequent request once the previous response is received.
• To simulate multiple end users using the service simultaneously, increase the request concurrency number to generate more loads to the NVIDIA Triton server.

While the NVIDIA Triton server from the previous step is still running, open a new terminal, connect using SSH to the instance that you were running, and run the NGC NVIDIA Triton SDK container:

docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:21.04-py3-sdk 

Generate a real input file with the format required. For an easy start, you can also directly download one example available input.json.

Run the perf analyzer with the following command:

perf_analyzer -m bert_base_qa --input-data input.json 

This starts the perf analyzer sending the request with the default request concurrency 1 and batch size 1. A detailed log of throughput and latency with breakdown is printed for further analysis. You may also add the --concurrency-range and -b flags to increase the request concurrency and batch size to simulate more heavy load scenarios.  For example:

perf_analyzer -m bert_base_qa --input-data input.json --concurrency-range 8 -b 1 

The preceding command sends the request with request concurrency 8 and batch size 1 to the NVIDIA Triton server.

Tables 2 and 3 show the inference throughput and latency results.

Table 3 shows that NVIDIA Triton can provide higher throughput with the concurrent model execution and dynamic batching features compared to the baseline without these optimizations on the same infrastructure.

With the default configuration for BERT-Base, you can reduce P99 latency down to 2.5 ms. You can achieve a throughput of 1639 sentences/sec with P99 latency around 5ms and 1794 sentences/sec with P99 latency less than 10ms by combining dynamic batching and concurrent model execution.

With dynamic batching and concurrent model execution enabled, you can do inference for BERT-Large:

• A P99 latency of 4.8 ms with the lowest latency
• A best throughput of 449 sentences/sec under 10 ms P99 latency

Conclusion

To summarize, you converted a fine-tuned BERT model for QA tasks into a TensorRT engine, which is highly optimized for inference. The optimized BERT QA engine was then deployed on NVIDIA Triton Inference Server, with concurrent model execution and dynamic batching to get the best performance from NVIDIA T4 GPUs.

Be sure to visit NGC, where you can find GPU-optimized AI, high-performance computing (HPC), and data analytics applications, as well as enterprise-grade containers, pretrained AI models, and industry-specific SDKs to aid in the development of your own workload. Also, stay tuned for the upcoming TensorRT 8, which includes new features like sparsity optimization for NVIDIA Ampere Architecture GPUs, quantization-aware training, and an enhanced compiler to accelerate transformer-based networks.