The NVIDIA CloudXR platform will be available on NVIDIA GPU-powered virtual machine instances on Azure.
NVIDIA is raising the bar for XR streaming. Announced today, the NVIDIA CloudXR platform will be available on NVIDIA GPU-powered virtual machine instances on Azure.
NVIDIA CloudXR is built on NVIDIA RTX GPUs to enable streaming of immersive AR, VR or mixed reality experiences from anywhere. By streaming from the cloud, enterprises can easily set up and scale immersive experiences from any location, to any VR or AR device. Companies and customers no longer need to purchase expensive high-performing workstations to experience high-quality, immersive environments.
Delivering High-Fidelity Experiences from the Cloud
NVIDIA partners Innoactive, TechViz, and Vection Technologies are among the first to use CloudXR on Azure instances.
With NVIDIA CloudXR, customers of Vection Technologies can access the Mindesk platform from anywhere, using standalone AR or VR headsets like the HTC Vive Focus. Enterprises can now easily share their 3D CAD, CAM, and BIM environments among colleagues and customers in real time. And since most of the computation happens in the cloud, any business can take advantage of the capabilities of CloudXR without spending money on expensive computational powerhouses.
“One of the greatest obstacles to the adoption of AR and VR was the necessity for a dedicated GPU. And even when that is available, it is not enough for more complex and dynamic content like CAD, CAM, BIM and simulations,” said Gabriele Sorrento, director of Vection Technologies. “NVIDIA CloudXR solves both problems. By decoupling the app from the workstation, CloudXR makes complex AR and VR data available anywhere.”
Vection Technologies plans to ship the Mindesk Suite with NVIDIA CloudXR.
Another early user of CloudXR on Azure is Innoactive, a company that provides a platform to deploy VR training at scale in large organizations. Working with companies such as SAP, Innoactive uses NVIDIA CloudXR and Azure to enhance VR training while keeping content secure and easily accessible from the cloud.
“SAP customers have large and complex CAD models which they want to use in their VR process simulations and training applications,” said Michael Spiess, venture lead for extended reality at SAP. “With CloudXR, we can easily stream those VR simulations with the highest quality visuals to any device — whether it’s an HMD, a tablet, or even a smartphone. Innoactive Portal supports NVIDIA CloudXR and makes all this available with a click of a button. This will help us to offer XR Cloud services to our 200 million business users.”
Users can try CloudXR on Azure today by downloading the Innoactive Portal app, and experience XR streaming from the cloud with their own VR headset.
CloudXR Sets the Stage for Advanced AR and VR
The CloudXR platform includes:
NVIDIA CloudXR SDK, which supports all OpenVR apps and includes broad client support for phones, tablets and head-mounted displays.
NVIDIA RTX Virtual Workstations to provide high quality graphics at the fastest frame rates.
NVIDIA AI SDKs to accelerate performance and increase immersive presence.
With CloudXR running on GPU-powered VM instances on Azure, companies can provide users with high-quality virtual experiences from anywhere in the world.
Availability Coming Soon
NVIDIA CloudXR on Azure will be generally available later this year, with a private beta available soon. Apply now for early access to the SDK, and sign up to get the latest news and updates on upcoming CloudXR releases, including the private beta.
This DevKit targets heterogeneous GPU/CPU system development, and includes an Arm CPU, an NVIDIA A100 Tensor Core GPU, and the NVIDIA HPC SDK suite of tools.
NVIDIA announced the NVIDIA Arm HPC Developer Kit, an integrated hardware-software platform for creating, evaluating, and benchmarking HPC, AI and scientific computing applications. Developers can apply today for the early interest program.
This DevKit targets heterogeneous GPU/CPU system development, and includes an Arm CPU, an NVIDIA A100 Tensor Core GPU, and the NVIDIA HPC SDK suite of tools.
It delivers:
A validated system for quick and easy bring-up in a stable environment for accelerated computing code execution and evaluation, performance analysis, system experimentation, and system characterization.
A stable hardware and software platform for development and performance analysis of accelerated HPC, AI, and scientific computing applications
Experimentation and characterization of high-performance, NVIDIA-accelerated, Arm server-based system architectures
The NVIDIA Arm HPC Developer Kit is based on the GIGABYTE G242-P32 2U server, and leverages the NVIDIA HPC SDK, a comprehensive suite of compilers, libraries, and tools for HPC delivering performance, portability, and productivity. The platform will support Ubuntu, SLES, and RHEL operating systems.
Developers are invited to apply for our early interest program to stay informed of pricing and availability. Learn more and apply now.
AI machine learning is unlocking breakthrough applications in fields such as online product recommendations, image classification, chatbots, forecasting, and manufacturing quality inspection. There are two parts to AI: training and inference. Inference is the production phase of AI. The trained model and associated code are deployed in the data center or public cloud, or at … Continued
AI machine learning is unlocking breakthrough applications in fields such as online product recommendations, image classification, chatbots, forecasting, and manufacturing quality inspection. There are two parts to AI: training and inference.
Inference is the production phase of AI. The trained model and associated code are deployed in the data center or public cloud, or at the edge to make predictions. This process is called inference serving and is complex for the following reasons:
Multiple model frameworks: Data scientists and researchers use different AI and deep learning frameworks like TensorFlow, PyTorch, TensorRT, ONNX Runtime, or just plain Python to build models. Each of these frameworks requires an execution backend to run the model in production.
Different inference query types: Inference serving requires handling different types of inference queries like real time online predictions, offline batch, streaming data, and a complex pipeline of multiple models. Each of these requires special processing for inference.
Constantly evolving models: Models are continuously retrained and updated based on new data and new algorithms. So, the models in production need to be updated continuously without restarting the server or any downtime. A given application can use many different models and it compounds the scale.
Diverse CPUs and GPUs: The models can be executed on a CPU or GPU and there are different types of GPUs and CPUs.
Often, organizations end up having multiple, disparate inference serving solutions, per model, per framework, or per application.
NVIDIA Triton Inference Server is an open-source inference serving software that simplifies inference serving for an organization by addressing the above complexities. Triton provides a single standardized inference platform which can support running inference on multi-framework models, on both CPU and GPU, and in different deployment environments such as datacenter, cloud, embedded devices, and virtualized environments.
It natively supports multiple framework backends like TensorFlow, PyTorch, ONNX Runtime, Python, and even custom backends. It supports different types of inference queries through advanced batching and scheduling algorithms, supports live model updates, and runs models on both CPUs and GPUs. Triton is also designed to increase inference performance by maximizing hardware utilization through concurrent model execution and dynamic batching. Concurrent execution allows you to run multiple copies of a model, and multiple different models, in parallel on the same GPU. Through dynamic batching, Triton can dynamically group together inference requests on the server-side to maximize performance.
Figure 1. Triton Inference Server architecture.
Automatic model conversion and deployment
To further simplify the process of deploying models in production, the 2.9 release introduces a new suite of capabilities. A trained model is generally not optimized for deployment in production. You must go through a series of conversion and optimizations for your specific target environment. The following process shows a TensorRT deployment using Triton on a GPU. This applies to any inference framework, whether it’s deployed on CPU or GPU.
Convert models from different frameworks (TensorFlow, PyTorch) to TensorRT to get the best performance.
Check for optimizations, such as precision.
Generate the optimized model.
Validate the accuracy of the model post-conversion.
Manually test different model configurations to find the one that maximizes performance, such as batch size, number of concurrent model instances per GPU.
Prepare model configuration and repository for Triton.
(Optional) Prepare a Helm chart for deploying Triton on Kubernetes.
As you can see, this process can take significant time and effort.
You can use the new Model Navigator in Triton to automate the process. Available as an alpha version, it can convert an input model from any framework (TensorFlow and PyTorch are supported in this release) to TensorRT, validate the conversion for correctness, automatically find and create the most optimal model configuration, and generate the repo structure for the model deployment. What could take days for each type of model now can be done in hours.
Optimizing model performance
For efficient inference serving, you must identify the optimal model configurations, such as the batch size and number of concurrent models. Today, you must try different combinations manually to eventually find the optimal configuration for your throughput, latency, and memory utilization requirements.
Triton Model Analyzer is another optimization tool that automates this selection for you by automatically finding the best configuration for models to get the highest performance. You can specify performance requirements, such as a latency constraint, throughput target, or memory footprint. The model analyzer searches through different model configurations and finds the one that provides the best performance under your constraints. To help visualize the performance of the top configurations, it then outputs a summary report, which includes charts (Figure 2).
Figure 2. Choosing optimal configuration with Triton Model Analyzer.
You do not have to settle for a less optimized inference service because of the inherent complexity in getting to an optimized model. You can use Triton to easily get to the most efficient inference.
Triton giant model inference
Models are rapidly growing especially in areas of natural language processing. For example, consider the GPT-3 model. Its full capabilities are still being explored. It has been shown to be effective in use cases such as reading comprehension and summarization of text, Q&A, human-like chatbots, and software code generation.
In this post, we don’t delve into the models. Instead, we look at the cost-effective inference of such giant models. GPUs are naturally the right compute resource for these workloads, but these models are so large that they cannot fit on a single GPU. For more information about a framework to train giant models on multi-GPU, multi-node systems, see Scaling Language Model Training to a Trillion Parameters Using Megatron.
Figure 3. Exponential growth in the size of language models.
For inference, you must split the model into multiple smaller files and load each on a separate GPU. There are generally two approaches to splitting a model:
Pipeline parallelism splits the model vertically across layer boundaries and runs these layers across multiple GPUs in a pipeline.
Tensor parallelism cuts the network horizontally and splits individual layers across GPUs.
The Triton custom backend feature can be used to run multi-GPU, multi-node backends. In addition, Triton has a model ensemble feature that can be used for pipeline parallelism.
Figure 4. Tensor model and pipeline parallelism.
Giant models are still in their infancy, but it won’t be long until you see production AI running multiple giant models for different use cases.
Ecosystem integrations
Inference serving is a production activity and can require integration with many ecosystem software and tools. Triton integrates with several, with new integrations added regularly.
Framework backends: Triton supports all the major deep learning framework execution backends out of the box, like TensorFlow, PyTorch, and ONNX RT. It allows custom backends in C++ and Python to be integrated easily. As part of the 21.03 release, a beta version of the OpenVINO backend in Triton is available for high performance CPU inferencing on the Intel platform.
Kubernetes ecosystem: Triton is designed for scale. It integrates with major Kubernetes platforms on the cloud like AKS, EKS, and GKE and on-premises implementations including Red Hat OpenShift. It is also part of the Kubeflow project.
Cloud AI platforms and MLOPs software: MLOps is a process that brings automation and governance to AI development and deployment workflow. Triton is integrated in cloud AI platforms such as Azure ML and can be deployed in Google CAIP and Amazon SageMaker as a custom container. It is also integrated with KFServing, Seldon Core, and Allegro ClearML.
Customer case studies
Triton is used by both large and small customers to serve models in production for all types of applications, such as computer vision, natural language processing, and recommender systems. Here are a few examples across different use cases:
Volkswagen Smart Lab: Integrates Triton into its Volkswagen Computer Vision Workbench so users can make contributions to the Model Zoo without needing to worry about whether they are based on ONNX, PyTorch, or TensorFlow frameworks. Triton simplifies model management and deployment, and that’s key for VW’s work serving up AI models in new and interesting environments, Bormann says in his GTC session, Taming the Computer Vision Zoo with NVIDIA Triton for an easy scalable eco system.
Natural language processing
Salesforce: Uses Triton to develop state-of-the-art transformer models. For more information, see the Benchmarking Triton Inference Server GTC 21 session.
LivePerson: Uses Triton as a standardized platform to serve NLP models from different frameworks on both CPUs and GPUs for chatbot applications.
Ant Group: Uses Triton as a standardized platform to serve models for different applications, such as AntChain, copyright management platform, fraud detection, and more.
Figure 5. Triton for standardized AI deployment across the data center.
Conclusion
Triton helps with a standardized scalable production AI in every data center, cloud, and embedded device. It supports multiple frameworks, runs models on both CPUs and GPUs, handles different types of inference queries, and integrates with Kubernetes and MLOPs platforms.
Volvo Cars is extending its long-held legacy of safety far into the future. The global automaker announced during the GTC keynote today that it will use NVIDIA DRIVE Orin to power the autonomous driving computer in its next-generation cars. The decision deepens the companies’ collaboration to even more software-defined model lineups, beginning with the next-generation Read article >
Developing autonomous vehicles with large scale simulation requires an ecosystem of partners and tools that’s just as wide ranging. NVIDIA DRIVE Sim powered by Omniverse addresses AV development challenges with a scalable, diverse and physically accurate simulation platform. With DRIVE Sim, developers can improve productivity and test coverage, accelerating their time to market while minimizing Read article >
Carestream Health, a leading maker of medical imaging systems, is investigating the use of NVIDIA Clara AGX — an embedded AI platform for medical devices — in the development of AI-powered features on single-frame and streaming x-ray applications. Startups around the world, too, are adopting Clara AGX for AI solutions in medical imaging, surgery and Read article >
The NVIDIA HPC SDK is a comprehensive suite of compilers, libraries, and tools enabling developers to program the entire HPC platform from the GPU foundation to the CPU, and through the interconnect.
Today, NVIDIA is announcing the availability of the HPC SDK version 21.3. This software can be downloaded now free of charge.
The NVIDIA HPC SDK is a comprehensive suite of compilers, libraries, and tools enabling developers to program the entire HPC platform from the GPU foundation to the CPU, and through the interconnect. It is the only comprehensive, integrated SDK for programming accelerated computing systems.
The NVIDIA HPC SDK C++ and Fortran compilers are the first compilers to support automatic GPU acceleration of standard language constructs including C++17 parallel algorithms and Fortran intrinsics.
Learn more:
GTC 2021: S31286 A Deep Dive into the Latest HPC Software — GTC is Free, Register Now
Data plays a crucial role in creating intelligent applications. To create an efficient AI/ ML app, you must train machine learning models with high-quality, labeled datasets. Generating and labeling such data from scratch has been a critical bottleneck for enterprises. Many companies prefer a one-stop solution to support their AI/ML workflow from data generation, data … Continued
Data plays a crucial role in creating intelligent applications. To create an efficient AI/ ML app, you must train machine learning models with high-quality, labeled datasets. Generating and labeling such data from scratch has been a critical bottleneck for enterprises. Many companies prefer a one-stop solution to support their AI/ML workflow from data generation, data labeling, model training/fine-tuning, and deployment.
To fast track the end-to-end workflow for developers, NVIDIA has been working with several partners who focus on generating large, diverse, and high-quality labeled data. Their platforms can be seamlessly integrated with NVIDIA Transfer Learning Toolkit (TLT) and NeMo for training and fine-tuning models. These efficiently trained and optimized models can then be deployed with NVIDIA DeepStream or NVIDIA Jarvis to create reliable computer vision or conversational AI applications.
Figure 1. End-to-end workflow of training a model and deploying AI applications.
In this post, we outline the key challenges in data preparation and training. We also introduce how to integrate your data to fine-tune AI/ML models easily with our partner services.
Computer vision
Large amounts of labeled data are needed to train computer vision neural network models. They could be collected from the real world or synthesized through models. High-quality labeled data enables neural network models to contextualize the information and generate accurate results.
NVIDIA integrated with the platforms from the following partners to generate synthetic and label custom data in the formats compatible with TLT for training. TLT is a zero-coding transfer learning toolkit using purpose-built, pretrained models. Trained TLT models can be deployed using DeepStream SDK and achieve 10x speedup in development time.
AI Reverie and Sky Engine
Synthetic labeled data is becoming popular especially for computer vision tasks like object detection and image segmentation. Using the platforms from AI Reverie and Sky Engine, you can generate synthetic labeled data. AI Reverie offers a suite of synthetic data for model training and validation using 3D environments which exposes the neural network models to diverse scenarios that might be hard to find in data gleaned from the real world. Sky Engine uses ray-tracing image renderer techniques to generate labeled synthetic data in virtual environments that can be used with TLT for training.
Figure 2. Computer vision workflow from unlabeled dataset to application deployment.
Appen
With Appen, you can generate and label custom data. Appen uses APIs and human intelligence to generate labeled training data. Integrated with the NVIDIA Transfer Learning Toolkit, the Appen Data Annotation platform and services allow you to eliminate time-consuming annotations and create the right training data to train with TLT for your use cases.
Hasty, Labelbox, and Sama
Few provide straightforward tools for annotation. By simple clicks and selecting the region around the objects on the image, you can quickly generate the annotations. With Hasty, Sama, and Labelbox, you can label datasets in the formats compatible with TLT. TLT is a zero-coding transfer learning toolkit using purpose-built, pretrained models. Trained TLT models can be deployed using the DeepStream SDK and achieve 10x speedup in development time.
Hasty provides such tools like DEXTR and GrabCut to create labeled data. With DEXTR, you click on the north, south, east, and west of an object and a neural network looks for the mask. For GrabCut, you select the area where the object is and add or remove regions with markings to improve the results.
With Labelbox, you can upload the data for annotation and can easily export the annotated data using Python SDK to TLT for training purposes.
Sama follows a different labeling mechanism and therefore uses a pretrained object detection model (Figure 2) to perform inference. IT uses those annotations to further enhance the labels in generating accurate results.
All these tools are tightly coupled with TLT for training to develop production-quality vision models.
Conversational AI
A typical conversational AI application may be built with automatic speech recognition (ASR), natural language understanding (NLU), dialog management (DM) and text-to-speech (TTS) services. Conversational AI models are huge and require large amounts of data for training to recognize speech in noisy environments in different languages or accents, understand every nuance of language and generate human-like speech.
NVIDIA collaborates with DefinedCrowd, LabelStudio, and DataSaur. You can use their platforms to create large and labeled datasets that can then be directly introduced into NVIDIA NeMo for training or fine-tuning. NVIDIA NeMo is an open-source toolkit for developing state-of-the-art conversational AI models.
DefinedCrowd
The popular approach for data generation/labeling is crowdsourcing. DefinedCrowd leverages a global crowd of over 500,000 contributors to collect, annotate, and validate training data by dividing the project into a series of micro-tasks through the online platform Neevo. The data is then used to train conversational AI models in a variety of languages, accents, and domains. You can now quickly download the data using the DefinedCrowd APIs and convert them into NeMo acceptable format for training.
LabelStudio
Figure 3. Speech recognition workflow from unlabeled dataset to application deployment.
Advancement of machine learning models make AI-assisted data annotation possible, which significantly reduces labeling cost. More and more vendors are integrating AI capability to augment the labeling process. NVIDIA collaborated with Heartex who maintains the open-source LabelStudio and performs AI-assisted data annotation for speech. With this tool, you can now label speech data that is compatible with NeMo (Figure 3). Heartex also integrates NeMo into their workflow so that you can choose the pretrained NeMo models for speech pre-annotation.
Datasaur
For text labeling, there are easy-to-use tools that come with predefined labels which allow you to quickly annotate the data. For more information about how to annotate your text data and run Datasaur with NeMo for training, see Datasaur x NVIDIA NeMo Integration (video).
Conclusion
NVIDIA has partnered with many industry-leading data generation and annotation providers to speed up computer vision and conversational AI application workflows. This collaboration enables you to quickly generate superior quality data that can be readily used with TLT and NeMo for training models, which can then be deployed on NVIDIA GPUs with DeepStream SDK and Jarvis.
For more information, see the following resources:
SANTA CLARA, Calif., April 12, 2021 (GLOBE NEWSWIRE) — NVIDIA today announced at its annual Investor Day that first quarter revenue for fiscal 2022 is tracking above its previously provided …
The NVIDIA CloudXR platform will be available on NVIDIA GPU-powered virtual machine instances on Azure.
This DevKit targets heterogeneous GPU/CPU system development, and includes an Arm CPU, an NVIDIA A100 Tensor Core GPU, and the NVIDIA HPC SDK suite of tools. 
AI machine learning is unlocking breakthrough applications in fields such as online product recommendations, image classification, chatbots, forecasting, and manufacturing quality inspection. There are two parts to AI: training and inference. Inference is the production phase of AI. The trained model and associated code are deployed in the data center or public cloud, or at … 
Catch the latest announcements from NVIDIA CEO Jensen Huang
The NVIDIA HPC SDK is a comprehensive suite of compilers, libraries, and tools enabling developers to program the entire HPC platform from the GPU foundation to the CPU, and through the interconnect. 
Data plays a crucial role in creating intelligent applications. To create an efficient AI/ ML app, you must train machine learning models with high-quality, labeled datasets. Generating and labeling such data from scratch has been a critical bottleneck for enterprises. Many companies prefer a one-stop solution to support their AI/ML workflow from data generation, data … 
