The Acceleration Agency, a digital innovation and product design firm, is working on an active digital twin framework and toolkit called Project Gemini….
The Acceleration Agency, a digital innovation and product design firm, is working on an active digital twin framework and toolkit called Project Gemini. Inspired by the United States space program of the same name, Project Gemini uses active sensor fabric data and a wide range of data from sources like Google Sheets and Customer Relationship Management (CRM) platforms to replicate real-world settings in the virtual world.
The project launched with a digital replication of The Acceleration Agency’s main office located in Austin, Texas. Instrumented with a dense sensor fabric for real-time and historical spatial computation, the digital twin of the office includes employees and employee information (job title, ID#, gender, and date of birth) provided by Salesforce. It also tracks inventory items on site and can display information such as quantity, date of last interaction, temperature, and orientation.
With NVIDIA Omniverse real time, true-to-reality physics from PhysX, and physically accurate RTX rendering capabilities, the team anticipates that the Gemini active digital twin can be simulated with an unprecedented level of visual and physical fidelity and with complex simulations.
Leveraging USD and Omniverse Nucleus, users of the Project Gemini digital twin platform will be able to update content in a variety of tools in real time collaboratively instead of having to wait for new builds.
Connecting Google Sheets to NVIDIA Omniverse with a Kit Extension
Multiple abstraction layers and a sensor fabric layer allow a variety of sensors, databases, CRMs and object integration tools to connect to Omniverse. The connection allows real-time updates to inventory objects and information like temperature, humidity, and location.
To accomplish this, the team created a simple Omniverse Kit Extension enabled by a Python script that reads data from a Google Sheet and attaches the data to an object in Omniverse Kit. It allows someone to control the location, scale, and rotation of any selected object in Omniverse applications like Omniverse Code or Omniverse Create using the metadata in the spreadsheet. You can access the AccelerationAgency/omniverse-extensions through GitHub.
Using database and CRM tools with the extension makes the task of manipulating object data more scalable. When building digital twins at the scale of factories, stadiums, warehouses, and even cities, hundreds, thousands, and even millions of objects may need to be manipulated rapidly.
The Acceleration Agency loaded the USD version of their office digital twin into the Omniverse stage and used the extension to select and manipulate object data.
The images below show an example of how this process was done for a Tesla in the parking lot outside the agency office. Building this was fairly straightforward and only took a few days for a single developer to create. It can be extended to any data source.
Figure 1. Google Sheet with object location, scale, and rotation information
Figure 2. Selecting the Project Gemini-enabled extension from the extensions tab in Omniverse Code
Figure 3. The object before running the extension to pull in the data from the Google Sheet
Figure 4. After running the extension to pull in the data from the Google Sheet, the object now has different parameters
Figure 5. Running the extension using the USD version of the office digital twin as the data source, then selecting the Tesla as the data object to manipulate
Figure 6. Tripling the scale factors of the Tesla in the Google Sheet updates through the extension and then propagates into the stage
Watch the extension in action with Starr Long, Executive Producer at The Acceleration Agency:
Adding RTX Renderer and Nucleus Collaboration
The next step for Project Gemini is to render in real time with the NVIDIA RTX Renderer and allow for real-time modifications through Nucleus. The real-time modifications are one of the advantages of working with the powerful USD 3D framework and composition engine. This will be coupled with historical recordings of real data which when played back can be mixed with these modifications to try different scenarios. Some of the use cases the team is targeting include construction sites, hospitals, and live event venues. To learn more, visit the Project Gemini website.
Figure 7. Digital twin of The Acceleration Agency office running in the NVIDIA RTX Renderer
Figure 8. Sensors and tags that send real-time data about location, temperature, and other factors to the digital twin
You’re also invited to enter the inaugural #ExtendOmniverse developer contest, open through August 19, 2022. Create an Omniverse Extension using Omniverse Code for a chance to win an NVIDIA RTX GPU.
Innovative technologies in AI, virtual worlds and digital humans are shaping the future of design and content creation across every industry. Experience the latest advances from NVIDIA in all these areas at SIGGRAPH, the world’s largest gathering of computer graphics experts, running Aug. 8-11. At the conference, creators, developers, engineers, researchers and students will see Read article >
Imagine hiking to a lake on a summer day — sitting under a shady tree and watching the water gleam under the sun. In this scene, the differences between light and shadow are examples of direct and indirect lighting. The sun shines onto the lake and the trees, making the water look like it’s shimmering Read article >
Pinterest has engineered a way to serve its photo-sharing community more of the images they love. The social-image service, with more than 400 million monthly active users, has trained bigger recommender models for improved accuracy at predicting people’s interests. Pinterest handles hundreds of millions of user requests an hour on any given day. And it Read article >
It’s the first GFN Thursday of the month and you know the drill — GeForce NOW is bringing a big batch of games to the cloud. Get ready for 38 exciting titles like Saints Row and Rumbleverse arriving on the GeForce NOW library in August. Members can kick off the month streaming 13 new games Read article >
Starting today, the NVIDIA Jetson AGX Orin 32GB production module is available for purchase. Combined with the world-standard NVIDIA AI software stack and an…
Starting today, the NVIDIA Jetson AGX Orin 32GB production module is available for purchase. Combined with the world-standard NVIDIA AI software stack and an ecosystem of services and products, the road to market has never been faster.
The NVIDIA Jetson AGX Orin 32GB module delivers up to 200 trillion operations per second (TOPS) of AI performance with power configurable between 15W and 40W, which is more than 6x the performance of Jetson AGX Xavier in the same compact form-factor for robotics and other autonomous machine use cases.
This latest system-on-module supports multiple concurrent AI application pipelines with an NVIDIA Ampere architecture GPU, next-generation deep learning and vision accelerators, high-speed IO, and fast memory bandwidth. Developers can build solutions using their largest and most complex AI models to solve problems such as natural language understanding, 3D perception, and multi-sensor fusion.
Jetson runs the NVIDIA AI software stack, and use-case specific application frameworks are available, including NVIDIA Isaac for robotics, DeepStream for vision AI, and Riva for conversational AI.
You can also save time using the NVIDIA Omniverse Replicator for synthetic data generation (SDG), and with NVIDIA TAO Toolkit for fine-tuning pretrained AI models from the NGC catalog.
A broad range of Jetson ecosystem partners offer additional AI and system software, developer tools, and custom software development. They can also help with cameras and other sensors, as well as full system carrier boards and design services for your product.
Learn more about all the partners solutions supporting Orin that were announced today.
The Jetson AGX Orin Developer Kit is also available now. Use it to accelerate your Orin development, emulate the entire family of Jetson Orin modules, and create advanced robotics and edge AI applications.
Look for additional Jetson Orin modules later this year, including Jetson Orin NX 16GB in October, Jetson AGX Orin 64GB in November, and Jetson Orin NX 8GB in December. Sign up to be notified when new Jetson Orin modules become available.
This is the second part of a two-part series about NVIDIA tools that allow you to run large transformer models for accelerated inference. For an introduction to…
This post is a guide to optimized inference of large transformer models such as EleutherAI’s GPT-J 6B and Google’s T5-3B. Both of these models demonstrate good results in many downstream tasks and are among the most available to researchers and data scientists.
NVIDIA FasterTransformer (FT) in NVIDIA Triton allows you to run both of these models in a similar and simple manner while providing enough flexibility to integrate/combine with other inference or training pipelines. The same NVIDIA software stack can be used for inference of the trillion-parameters models combining tensor parallelism (TP) and pipeline parallelism (PP) techniques on multiple nodes.
Transformer models are increasingly used in numerous domains and demonstrate outstanding accuracy. More importantly, the size of the model directly affects its quality. Apart from the NLP, this is applicable to other domains as well.
Researchers from Google demonstrated that the scaling of the transformer-based text encoder was crucial for the whole image generation pipeline in their Imagen model, the latest and one of the most promising generative text-to-image models. Scaling the transformers leads to outstanding results in both single and multi-domain pipelines. This guide uses transformer-based models of the same structure and a similar size.
Overview and walkthrough of main steps
This section presents the main steps for running T5 and GPT-J in optimized inference using FasterTransformer and Triton Inference Server. Figure 1 demonstrates the overall process for one neural network.
It is highly recommended to do all the steps in a Docker container to reproduce the results. Instructions about preparing a FasterTransformer Docker container are available at the beginning of the same notebook.
If you have pretrained one of these models, you will have to convert the weights from your framework saved-model files into the binary format recognizable by the FT. Scripts for conversion are provided in the FasterTransformer repository.
Figure 1. An overall pipeline of the transformer neural network with FasterTransformer and Triton
Steps 1 and 2: Build Docker container with Triton inference server and FasterTransformer backend. Use the Triton inference server as the main serving tool proxying requests to the FasterTransformer backend.
Steps 3 and 4: Build the FasterTransformer library. This library contains many useful tools for inference preparation as well as bindings for multiple languages and examples of how to do inference in C++ and Python.
Steps 5 and 6: Download weights of the pretrained models (T5-3B and GPT-J) and prepare them for the inference with FT by converting into binary format and splitting them into multiple partitions for parallelism and accelerated inference. Code from the FasterTransformer library will be used in this step.
Step 7: Use code from the FasterTransformer library to find optimal low-level kernels for the NN.
Step 8: Start the Triton server that uses all artifacts from previous steps and run the Python client code to send requests to the server with accelerated models.
Step 1: Clone fastertransformer_backend from the Triton GitHub repository
All further steps need to be run inside the Docker container interactive session. Jupyter Lab is also needed in this container to work with the notebook provided.
The Docker container was built with Triton and FasterTransformer and started with the fastertransformer_backend source codes inside.
Steps 3 and 4: Clone FasterTransformer source codes and build the library
The FasterTransformer library is pre-built and placed into our container during the Docker build process.
Download the FasterTransformer source code from GitHub to use the additional scripts that allow converting the pre-trained model files of the GPT-J or T5 into FT binary format that will be used at the time of inference.
GPT-J is a decoder model that was developed by EleutherAI and trained on The Pile, an 825GB dataset curated from multiple sources. With 6 billion parameters, GPT-J is one of the largest GPT-like publicly-released models.
FasterTransformer backend has a config for the GPT-J model under fastertransformer_backend/all_models/gptj. This config is a perfect demonstration of a Triton ensemble. Triton allows you to run a single model inference, as well as construct complex pipes/pipelines comprising many models required for an inference task.
You can also add additional Python/C++ scripts before and/or after any neural network for pre/post processing steps that could transform your data/results into the final form.
The GPT-J inference pipeline includes three different sequential steps at the server side:
The config file combines all three stages into a single pipeline. Figure 2 illustrates the client-server inference scheme.
Figure 2. GPT-J inference with FasterTransformer and Triton. Scheme of the ensemble with all pre- and post-processing steps happening on the server side
Steps 5-8 are the same for both GPT-J and T5 and are provided below (GPT first, followed by T5).
Step 5 (GPT-J): Download and prepare weights of the GPT-J model
wget https://mystic.the-eye.eu/public/AI/GPT-J-6B/step_383500_slim.tar.zstd
tar -axf step_383500_slim.tar.zstd -C ./models/
These weights need to be converted into the binary format recognized by the C++ FasterTransformer backend. FasterTransformer provides the tools/scripts for different pretrained neural networks.
For the GPT-J weights you can use the script:
FasterTransformer/examples/pytorch/gptj/utils/gptj_ckpt_convert.py to convert the checkpoint as follows:
The n-inference-gpus specifies the number of GPUs for tensor parallelism. This script will create ./models/j6b_ckpt/2-gpu directory and automatically write prepared weights there. These weights will be ready for TensorParallel 2 inference. Using this parameter, you can split your weights onto a larger number of GPUs to achieve even higher speed using the TP technique.
Step 7 (GPT-J): Kernel-autotuning for the GPT-J inference
The next step is kernel-autotuning. Matrix multiplication is the main and the heaviest operation in transformer-based neural networks. FT uses functionalities from CuBLAS and CuTLASS libraries to execute this type of operation. It is important to note that MatMul operation can be executed in tens of different ways using different low-level algorithms at the “hardware” level.
The FasterTransformer library has a script that allows real-time benchmarking of all low-level algorithms and selection of the best one for the parameters of the model (size of the attention layers, number of attention heads, size of the hidden layer) and for your input data. This step is optional but achieves a higher inference speed.
Run the ./FasterTransformer/build/bin/gpt_gemm binary file that was built at the stage of building FasterTransformer library. Arguments for the script may be found in the GitHub’s documentation or by using --help argument.
Step 8 (GPT-J): Prepare the Triton config and serve the model
With the weights ready, the next step is to prepare a Triton config file for the GPT-J model. Open the main Triton config for the GPT-J model at fastertransformer_backend/all_models/gptj/fastertransformer/config.pbtxt for editing. Only two mandatory parameters need to be changed there to start inference.
Update tensor_para_size. Weights were prepared for two GPUs, so set it equal to 2.
If Triton starts successfully, you will see output lines informing that the models are loaded by Triton and the server is listening the designated ports for incoming requests:
# Info about T5 model that was found by the Triton in our directory:
+-------------------+---------+--------+
| Model | Version | Status |
+-------------------+---------+--------+
| fastertransformer | 1 | READY |
+-------------------+---------+--------+
# Info about that Triton successfully started and waiting for HTTP/GRPC requests:
I0503 17:26:25.226719 1668 grpc_server.cc:4421] Started GRPCInferenceService at 0.0.0.0:8001
I0503 17:26:25.227017 1668 http_server.cc:3113] Started HTTPService at 0.0.0.0:8000
I0503 17:26:25.283046 1668 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002
Next, send the inference requests to the server. On the client side, the tritonclient Python library allows communicating with our server from any of the Python apps.
This example with GPT-J sends textual data straight to the Triton server and all preprocessing and postprocessing will happen on the server side. The full client script can be found at fastertransformer_backend/tools/end_to_end_test.py or in the Jupyter notebook provided.
The main parts include:
# Import libraries
import tritonclient.http as httpclient
# Initizlize client
client = httpclient.InferenceServerClient("localhost:8000",
concurrency=1,
verbose=False)
# ...
# Request text promp from user
print("Write any input prompt for the model and press ENTER:")
# Prepare tokens for sending to the server
inputs = prepare_inputs( [[input()]])
# Sending request
result = client.infer(MODEl_GPTJ_FASTERTRANSFORMER, inputs)
print(result.as_numpy("OUTPUT_0"))
T5 inference
T5 (Text-to-Text Transfer Transformer) is a recent architecture created by Google. It consists of encoder and decoder parts and is an instance of a full transformer architecture. It reframes all the natural language processing (NLP) tasks into a unified text-to-text format where the input and output are always text strings.
The T5 inference pipeline prepared in this section differs from the GPT-J model, in that only the NN inference stage is on the server side, not a full pipeline with data preprocessing and results in postprocessing. All computations for the pre- and post-processing stages are happening on the client side.
Triton allows you to configure your inference flexibly so it is possible to build a full pipeline on the server side too, but other configurations are also possible.
First, do a conversion from text into tokens in Python using the Huggingface library on the client side. Next, send an inference request to the server. Finally, after getting a response from the server, convert the generated tokens into text on the client side.
Figure 3 illustrates the client-server inference scheme.
Figure 3. T5 inference with FasterTransformer and Triton. All pre- and post-processing steps happen on the client side and only the heavy inference part computes is on the server.
Preparation steps for T5 are the same as for GPT-J. Details for steps 5-8 are provided below for T5:
Step 5 (T5): Download weights of the T5-3B
First download weights of the T5 3b size. You will have to install git-lfs to successfully download the weights.
git clone https://huggingface.co/t5-3b
Step 6 (T5): Convert weights into FT format
Again, the weights need to be converted into the binary format recognized by the C++ FasterTransformer backend. For T5 weights you can use the script at FasterTransformer/blob/main/examples/pytorch/t5/utils/huggingface_t5_ckpt_convert.py to convert the checkpoint.
The converter requires the following arguments. Quite similar to GPT-J but parameter i_g means the number of GPUs will be used for the inference in TP regime, so set it to 2:
Step 7 (T5): Kernel-autotuning for the T5-3B inference
The next step is kernel-autotuning for T5 using the t5_gemm binary file that will run experiments to benchmark the heaviest parts of the T5 model and find the best low-level kernels. Run ./FasterTransformer/build/bin/t5_gemm binary file that was built at the stage of building the FasterTransformer library (Step 2). This step is optional but including it achieves a higher inference speed. Again, the arguments for the script may be found in the GitHub’s documentation or by using --help argument.
Step 8 (T5): Prepare the Triton config of the T5 model
You will have to open the copied Triton config for the T5 model triton-model-store/t5/fastertransformer/config.pbtxt for editing. Only two mandatory parameters need to be changed there to start the inference.
Then update tensor_para_size. Weights were prepared for two GPUs, so set it to 2.
If Triton starts successfully, you will see these lines in the output:
# Info about T5 model that was found by the Triton in our directory:
+-------------------+---------+--------+
| Model | Version | Status |
+-------------------+---------+--------+
| fastertransformer | 1 | READY |
+-------------------+---------+--------+
# Info about that Triton successfully started and waiting for HTTP/GRPC requests:
I0503 17:26:25.226719 1668 grpc_server.cc:4421] Started GRPCInferenceService at 0.0.0.0:8001
I0503 17:26:25.227017 1668 http_server.cc:3113] Started HTTPService at 0.0.0.0:8000
I0503 17:26:25.283046 1668 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002
Now run the client script. On the client side, transform the textual input to tokens using the Huggingface library and only then send a request to the server using the Python’s tritonclient library. Implement function preprocessing for this purpose.
Then use an instance of the tritonclient http class that will request the 8000 port on the server (“localhost” if deployed locally) to send tokens to the model through HTTP.
After receiving the response containing the tokens, again transform the tokens into text form using a postprocessing helper function.
# Import libraries
from transformers import (
T5Tokenizer,
T5TokenizerFast
)
import tritonclient.http as httpclient
# Initialize client
client = httpclient.InferenceServerClient(
URL, concurrency=request_parallelism, verbose=verbose
)
# Initialize tokenizers from HuggingFace to do pre and post processings
# (convert text into tokens and backward) at the client side
tokenizer = T5Tokenizer.from_pretrained(MODEL_T5_HUGGINGFACE, model_max_length=1024)
fast_tokenizer = T5TokenizerFast.from_pretrained(MODEL_T5_HUGGINGFACE, model_max_length=1024)
# Implement the function that takes text converts it into the tokens using
# HFtokenizer and prepares tensorts for sending to Triton
def preprocess(t5_task_input):
...
# Implement function that takes tokens from Triton's response and converts
# them into text
def postprocess(result):
...
# Run translation task with T5
text = "Translate English to German: He swung back the fishing pole and cast the line."
inputs = preprocess(text)
result = client.infer(MODEl_T5_FASTERTRANSFORMER, inputs)
postprocess(result)
Adding custom layers and new NN architectures
If you have some custom neural network with the transformer blocks inside, or you have added some custom layers into default NNs supported by FT (T5, GPT), this NN won’t be supported by FT out-of-the-box. You can either change the source code of FT to add support for this NN by adding support for new layers, or you can use FT blocks and C++, PyTorch, and TensorFlow API to integrate fast transformer blocks from FT into your custom inference script/pipeline.
Results
The optimizations carried out by FasterTransformer achieved up to 6xspeed-up over native PyTorch GPU inference in FP16 mode and up to 33x speedup over PyTorch CPU inference for GPT-J and T5-3B.
Figure 4 shows the inference results for the GPT-J, and Figure 5 shows the inference results for T5-3B model at batch size 1 for the translation task.
Figure 4. GPT-J 6B model inference speed-up comparison
Figure 5. T5-3B model inference speed-up comparison
The smaller the model and the bigger the batch size, the better optimization FasterTransformer demonstrates due to increasing computational bandwidth. Figure 6 shows the T5-small model, tests of which can be found on the FasterTrasformer GitHub. It demonstrates a ~22x throughput increase in comparison with GPU PyTorch inference. Similar results can be found on GitHub for the T5-base model.
Figure 6. T5-small model inference comparison
Conclusion
The code example demonstrated here uses FasterTransformer and Triton inference server to run inference of the GPT-J-6B and T5-3B models. It achieved up to 33x acceleration in comparison with CPU and up to 22x in comparison with native PyTorch backend on GPU.
The same approach can be used for small transformer models like T5-small and BERT as well as huge models with trillions of parameters like GPT-3. Triton with FasterTransformer uses techniques like tensor and pipeline parallelism to provide optimized and highly accelerated inference to achieve low latency and high throughput for all of them.
Training and inference of large models is a non-trivial task on the edge between AI and HPC. If you are interested in huge neural networks, NVIDIA released multiple tools that can help you make the most of them in the easiest, most efficient way.
NeMo Megatron is a new capability in the NeMo framework that allows developers to effectively train and scale language models to billions of parameters. The inferencing relies on the same tools presented in this post. Learn more with Model Parallelism: Building and Deploying Large Neural Networks, a hands-on interactive live course on theoretical and practical aspects in training and inferencing of large models.
This is the first part of a two-part series discussing the NVIDIA FasterTransformer library, one of the fastest libraries for distributed inference of…
This is the first part of a two-part series discussing the NVIDIA FasterTransformer library, one of the fastest libraries for distributed inference of transformers of any size (up to trillions of parameters). It provides an overview of FasterTransformer, including the benefits of using the library.
Transformers are among the most influential AI model architectures today and are shaping the direction for future R&D in AI. Invented first as a tool for natural language processing (NLP), they are now used for almost any AI task, including computer vision, automatic speech recognition, classification of molecule structures, and processing of financial data. Accounting for such widespread use is the attention mechanism, which noticeably increases the computational efficiency, quality, and accuracy of the models.
Large transformer-based models with hundreds of billions of parameters behave like a gigantic encyclopedia and brain that contains information about everything it has learned. They structurize, represent, and summarize all this knowledge in a unique way. Having such models with this vast amount of prior knowledge allows us to use new and powerful one-shot or few-shot learning techniques to solve many NLP tasks.
Thanks to their computational efficiency, transformers scale well–and by increasing the size of the network and the amount of training data, researchers can improve observations and increase accuracy.
Training such large models is a non-trivial task, however. The models may require more memory than one GPU supplies–or even hundreds of GPUs. Thankfully, NVIDIA researchers have created powerful open-source tools, such as NeMo Megatron, that optimize the training process.
Fast and optimized inference allows enterprises to realize the full potential of these large models. The latest research demonstrates that increasing the size of the model as well as the dataset increases the quality of such a model on downstream tasks in different domains (NLP, CV, and others).
At the same time, data show that such a technique also works in multi-domain tasks. (See research papers like OpenAI’s DALLE-2 and Google’s Imagen on text-to-image generation, for example.) Research directions such as p-tuning that rely on “frozen” copies of huge models even increase the importance of having a stable and optimized inference pipeline. Optimized inference of such large models requires distributed multi-GPU multi-node solutions.
A library for accelerated inference of large transformers
NVIDIA FasterTransformer (FT) is a library implementing an accelerated engine for the inference of transformer-based neural networks, with a special emphasis on large models, spanning many GPUs and nodes in a distributed manner.
FasterTransformer contains the implementation of the highly-optimized version of the transformer block that contains the encoder and decoder parts.
Using this block, you can run the inference of both the full encoder-decoder architectures like T5, as well as encoder-only models, such as BERT, or decoder-only models, such as GPT. It is written in C++/CUDA and relies on the highly optimized cuBLAS, cuBLASLt , and cuSPARSELt libraries. This allows you to build the fastest transformer inference pipeline on GPU.
Figure 1. A couple of transformer/attention blocks are distributed between four GPUs using tensor parallelism (tensor MP partitions) and pipeline parallelism (pipeline MP partitions)
The distinctive feature of FT in comparison with other compilers like NVIDIA TensorRT is that it supports the inference of large transformer models in a distributed manner.
Figure 1 shows how a neural network with multiple classical transformer/attention layers could be split onto multiple GPUs and nodes using tensor parallelism (TP) and pipeline parallelism (PP) techniques.
Tensor parallelism occurs when each tensor is split up into multiple chunks, and each chunk of the tensor can be placed on a separate GPU. During computation, each chunk gets processed separately in-parallel on different GPUs and the results (final tensor) can be computed by combining results from multiple GPUs.
Pipeline parallelism occurs when a model is split up in-depth and different fulllayers are placed onto different GPUs/nodes.
Under the hood, enabling inter/intra-node communication relies on MPI and NVIDIA NCCL. Using this software stack, you can run large transformers in tensor parallelism mode on multiple GPUs to reduce computational latency.
At the same time, TP and PP may be combined together to run large transformer models with billions and trillions of parameters (which amount to terabytes of weights) on multi-GPU and multi-node environments.
Aside from the source codes in C, FasterTransformer also provides TensorFlow integration (using the TensorFlow op), PyTorch integration (using the PyTorch op), and Triton integration as a backend.
Currently, TensorFlow op only supports a single GPU, while PyTorch op and Triton backend both support multi-GPU and multi-node.
To prevent the additional work of splitting the model for model parallelism, FasterTransformer also provides a tool to split and convert models from different formats to the FasterTransformer binary file format. Then FasterTransformer can load the model in a binary format directly.
At this time, FT supports models like Megatron-LM GPT-3, GPT-J, BERT, ViT, Swin Transformer, Longformer, T5, and XLNet. You can check the latest support matrix in the FasterTransformer repo on GitHub.
FT works on GPUs with compute capability >= 7.0, such as V100, A10, A100, and others.
Figure 2. GPT-J 6B model inference speed-up comparison
Optimizations in FasterTransformer
FT enables you to get a faster inference pipeline, with lower latency and higher throughput for the transformer-based NNs in comparison to the common frameworks for deep learning training.
Some of the optimization techniques that allow FT to have the fastest inference for the GPT-3 and other large transformer models include:
Layer fusion – The set of techniques in the pre-processing stage that combine multiple layers of NNs into a single one that would be computed with one single kernel. This technique reduces data transfer and increases math density, thus accelerating computation at the inference stage. For example, all the operations in the multi-head attention block can be combined into one kernel.
Figure 3. Demonstration of the caching mechanism in the NVIDIA FasterTransformer library
Inference optimization for autoregressive models / activations caching
To prevent recomputing the previous keys and values for each new token generator by transformer, FT allocates a buffer to store them at each step.
Although it takes some additional memory usage, FT can save the cost of recomputing, allocating a buffer at each step, and the cost of concatenation. The scheme of the process is presented in Figure 2. The same caching mechanism is used in multiple parts of the NN.
Memory optimization
Different from traditional models like BERT, large transformer models have up to trillions of parameters taking hundreds of GB of storage. GPT-3 175b takes 350 GB even if we store the model in half-precision. It’s therefore necessary to reduce memory usage for other parts.
For example, in FasterTransformer, we reuse the memory buffer of activations/outputs in different decoder layers. Since the number of layers in GPT-3 is 96, we only need 1/96 of the amount of memoryfor activations.
Usage of MPI and NCCL to enable inter/intra-node communication and support model parallelism
In the GPT model, FasterTransormer provides both tensor parallelism and pipeline parallelism. For tensor parallelism, FasterTransformer follows the idea of Megatron. For both the self-attention block and feed-forward network block, FT split the weights of the first matrix by row and split the weights of the second matrix by column. By optimization, FT can reduce the reduction operation to two times for each transformer block.
For pipeline parallelism, FasterTransformer splits the whole batch of requests into multiple micro-batches, hiding the bubble of communication. FasterTransformer will adjust the micro-batch size automatically for different cases.
MatMul kernel autotuning (GEMM autotuning)
Matrix multiplication is the main and the heaviest operation in transformer-based neural networks. FT uses functionalities from CuBLAS and CuTLASS libraries to execute these types of operations. It is important to know that MatMul operation can be executed in tens of different ways using different low-level algorithms at the “hardware” level.
GemmBatchedEx function implements MatMul operation and has “cublasGemmAlgo_t” as an input parameter. Using this parameter, you can choose different low-level algorithms for operation.
The FasterTransformer library uses this parameter to do a real-time benchmark of all low-level algorithms and to choose the best one for the parameters of the model (size of the attention layers, number of attention heads, size of the hidden layer) and for your input data. Additionally, FT uses hardware-accelerated low-level functions for some parts of the network such as __expf, __shfl_xor_sync.
Inference with lower precisions
FT has kernels that support inference using low-precision input data in fp16 and int8. Both these regimes allow acceleration due to a lower amount of data transfer and required memory. At the same time, int8 and fp16 computations can be executed on special hardware, such as the tensor cores (for all GPU architectures starting from Volta), and the transformers engine in the upcoming Hopper GPUs.
More
Rapidly fast C++ BeamSearch implementation
Optimized all-reduce for the TensorParallelism 8 mode When the weights parts of the model are split between eight GPUs
NVIDIA Triton inference server with FasterTransformer backend
NVIDIA Triton Inference Server is an open-source inference serving software that helps standardize model deployment and execution, delivering fast and scalable AI in production. Stable and fast, Triton allows you to run inference of your ML/DL models in a simple manner with a pre-baked Docker container using only one line of code and a simple JSON-like config.
Triton supports models using multiple backends such as PyTorch, TorchScript, Tensorflow, ONNXRuntime, and OpenVINO. Triton takes your exported model that was trained in one of the frameworks and runs this model in inference using the corresponding backend transparently for you. It can also be extended with custom backends. Triton wraps your model with HTTP/gRPC API and provides client-side libraries for multiple languages.
Figure 4. Triton inference server with multiple backends for inference of model trained with different frameworks
Triton includes the FasterTransformer library as a backend (Figure 4) that enables running distributed multi-GPU, multi-node inference of large transformer models with TP and PP. Today, GPT-J, GPT-Megatron, and T5 models are supported in Triton with FasterTransformer backend.
Research on neural fields has been an increasingly hot topic in computer graphics and computer vision in recent years. Neural fields can represent 3D data like…
Research on neural fields has been an increasingly hot topic in computer graphics and computer vision in recent years. Neural fields can represent 3D data like shape, appearance, motion, and other physical quantities by using a neural network that takes coordinates as input and outputs the corresponding data at that location.
These representations have been proven to be useful in various applications like generative modeling and 3D reconstruction. NVIDIA projects such as NGLOD, GANcraft, NeRF-Tex, EG3D, Instant-NGP, and Variable Bitrate Neural Fields, are advancing state-of-the-art technology in neural fields, computer graphics, and computer vision in various ways.
Research challenges
Research on neural fields is moving fast, which means that standards and software often lag behind. Implementation differences can cause large variations in quality metrics and performance. The ramp-up cost for new projects can be considerable, with the components of neural fields increasing in complexity. Work is often duplicated among research groups–creating whole interactive applications to visualize the neural field outputs, for example.
One important milestone is NVIDIA Instant-NGP, which has recently attracted much attention from the research community due to its ability to fit various signals like neural radiance fields (NeRFs), signed distance fields (SDFs), and images at near-instant speeds. It unlocks a new frontier of practical applications and research directions due to its computational efficiency. However, this computational efficiency can also be a barrier for research due to the highly specialized and optimized code that can be difficult to adapt and extend.
NVIDIA Kaolin Wisp
NVIDIA Kaolin Wisp was developed as a fast-paced research-oriented library for neural fields to support researchers navigating the challenges of a growing discipline. It is built on top of the core Kaolin Library functionality, which includes more general and stable components for 3D deep learning research.
The goal of Wisp is to provide a common core library and framework for research on neural fields. The library consists of modular building blocks that can be used to create complex neural fields and an interactive app to train and visualize the neural fields.
Figure 1. A screenshot of the NVIDIA Kaolin Wisp interactive render, showing an optimization of a neural field in progress. The cameras and occupancy state of the occupancy structure are visualized on top. The properties inspector on the right allows users to derive more information about the scene and manipulate it.
Rather than providing a specific implementation, Wisp supplies the building blocks for neural fields. The framework is easily extensible for research purposes and consists of a modular pipeline where each pipeline component can be easily interchanged to provide a plug-and-play configuration for standard training.
Wisp does not aim to provide production-ready code, but to ship novel modules fast, staying at the leading edge of this technology. It also provides a rich set of examples that showcase the Kaolin Core framework and how Kaolin Core can be used to accelerate research.
Figure 2. NVIDIA Kaolin Wisp provides a variety of configurations and building blocks for researchers
NVIDIA Kaolin Wisp feature highlights
Kaolin Wisp uses a Python-based API, which builds on PyTorch, enabling users to develop a project quickly. Compatible with many other public PyTorch-based projects, Kaolin Wisp is easily customizable with PyTorch / CUDA-based building blocks.
While Wisp is designed for developer speed over compute performance, the building blocks that are provided in the library are optimized to train neural fields within minutes and visualize them interactively.
Kaolin Wisp is packed with building blocks to compose neural field pipelines with a mix-and-match approach. Notable examples are feature grids, which include:
Hierarchical Octrees: From NGLOD for learning features on spatial subdivision trees. The Octree also supports ray tracing operations, which allows for training a multiview image-based NGLOD-NeRF variant in addition to SDFs.
Triplanar Features: Used in EG3D and Convolutional Occupancy Networks papers to learn volumetric features on triplanar texture maps. The triplanes also support multiple levels of detail (LODs) in a multi-resolution pyramid structure.
Hash Grids: From the Instant-NGP paper for learning compact cache-friendly feature codebooks with performant memory access.
Figure 3. NVIDIA Kaolin Wisp architecture and building blocks
NVIDIA Kaolin Wisp is paired with an interactive renderer that supports flexible rendering of neural primitives pipelines, like variants of NeRF and neural SDFs. It allows the integration of new representations.
OpenGL style rasterized primitives can be mixed and matched with neural representations to add visualizations of more data layers, such as camera and occupancy structures. It also allows for easy-to-build customizable apps by supporting custom widgets on the GUI that can interact with the training and rendering.
Other useful features include property viewers, optimization controls, custom output render buffers, and camera object that allows for easy manipulation of scene cameras.
To learn more about Kaolin Wisp and other libraries, visit NVIDIA Research. You can access the kaolin-wisp project on GitHub.
Join NVIDIA 3D deep learning researchers and Kaolin library developers at SIGGRAPH 2022 for a session on Illuminating the Future of Graphics. Ask questions, watch demos, and learn how Kaolin Wisp can accelerate your neural network research.
The Acceleration Agency, a digital innovation and product design firm, is working on an active digital twin framework and toolkit called Project Gemini….
Join us September 19-22 for a deep dive into the latest advances in edge AI, from reimagined shopping experiences to industrial automation.
Starting today, the NVIDIA Jetson AGX Orin 32GB production module is available for purchase. Combined with the world-standard NVIDIA AI software stack and an…
This is the second part of a two-part series about NVIDIA tools that allow you to run large transformer models for accelerated inference. For an introduction to…
This is the first part of a two-part series discussing the NVIDIA FasterTransformer library, one of the fastest libraries for distributed inference of…
Research on neural fields has been an increasingly hot topic in computer graphics and computer vision in recent years. Neural fields can represent 3D data like…
