Categories
Misc

Upcoming Event: Recommender Systems Summit 2022

Join us to hear featured speakers from Netflix, Twitter, Weights & Biases, Coveo, and more discuss challenges building, training, optimizing, and deploying production-ready recommender systems.

Categories
Offsites

Minerva: Solving Quantitative Reasoning Problems with Language Models

Language models have demonstrated remarkable performance on a variety of natural language tasks — indeed, a general lesson from many works, including BERT, GPT-3, Gopher, and PaLM, has been that neural networks trained on diverse data at large scale in an unsupervised way can perform well on a variety of tasks.

Quantitative reasoning is one area in which language models still fall far short of human-level performance. Solving mathematical and scientific questions requires a combination of skills, including correctly parsing a question with natural language and mathematical notation, recalling relevant formulas and constants, and generating step-by-step solutions involving numerical calculations and symbolic manipulation. Due to these challenges, it is often believed that solving quantitative reasoning problems using machine learning will require significant advancements in model architecture and training techniques, granting models access to external tools such as Python interpreters, or possibly a more profound paradigm shift.

In “Solving Quantitative Reasoning Problems With Language Models” (to be released soon on the arXiv), we present Minerva, a language model capable of solving mathematical and scientific questions using step-by-step reasoning. We show that by focusing on collecting training data that is relevant for quantitative reasoning problems, training models at scale, and employing best-in-class inference techniques, we achieve significant performance gains on a variety of difficult quantitative reasoning tasks. Minerva solves such problems by generating solutions that include numerical calculations and symbolic manipulation without relying on external tools such as a calculator. The model parses and answers mathematical questions using a mix of natural language and mathematical notation. Minerva combines several techniques, including few-shot prompting, chain of thought or scratchpad prompting, and majority voting, to achieve state-of-the-art performance on STEM reasoning tasks. You can explore Minerva’s output with our interactive sample explorer!

Solving a multi-step problem: A question from the MATH dataset and Minerva’s solution. The model writes down a line equation, simplifies it, substitutes a variable, and solves for y.

A Model Built for Multi-step Quantitative Reasoning
To promote quantitative reasoning, Minerva builds on the Pathways Language Model (PaLM), with further training on a 118GB dataset of scientific papers from the arXiv preprint server and web pages that contain mathematical expressions using LaTeX, MathJax, or other mathematical typesetting formats. Standard text cleaning procedures often remove symbols and formatting that are essential to the semantic meaning of mathematical expressions. By maintaining this information in the training data, the model learns to converse using standard mathematical notation.

Example questions from the Joint Entrance Examination Main Math 2020 exam taken each year by almost 2M Indian high-school students intended to study engineering and similar fields (left), and the National Math Exam in Poland (May 2022) taken by approximately 270K high-school students every year (right).
A dataset for quantitative reasoning: Careful data processing preserves mathematical information, allowing the model to learn mathematics at a higher level.

Minerva also incorporates recent prompting and evaluation techniques to better solve mathematical questions. These include chain of thought or scratchpad prompting — where Minerva is prompted with several step-by-step solutions to existing questions before being presented with a new question — and majority voting. Like most language models, Minerva assigns probabilities to different possible outputs. When answering a question, rather than taking the single solution Minerva scores as most likely, multiple solutions are generated by sampling stochastically from all possible outputs. These solutions are different (e.g., the steps are not identical), but often arrive at the same final answer. Minerva uses majority voting on these sampled solutions, taking the most common result as the conclusive final answer.

Majority voting: Minerva generates multiple solutions to each question and chooses the most common answer as the solution, improving performance significantly.

Evaluation on STEM Benchmarks
To test Minerva’s quantitative reasoning abilities we evaluated the model on STEM benchmarks ranging in difficulty from grade school level problems to graduate level coursework.

  • MATH: High school math competition level problems
  • MMLU-STEM: A subset of the Massive Multitask Language Understanding benchmark focused on STEM, covering topics such as engineering, chemistry, math, and physics at high school and college level.
  • GSM8k: Grade school level math problems involving basic arithmetic operations that should all be solvable by a talented middle school student.

We also evaluated Minerva on OCWCourses, a collection of college and graduate level problems covering a variety of STEM topics such as solid state chemistry, astronomy, differential equations, and special relativity that we collected from MIT OpenCourseWare.

In all cases, Minerva obtains state-of-the-art results, sometimes by a wide margin.

Evaluation results on MATH and MMLU-STEM, which include high school and college level questions covering a range of STEM topics.
Model   MATH     MMLU-STEM     OCWCourses     GSM8k  
Minerva 50.3% 75% 30.8% 78.5%
Published state of the art    6.9% 55% 74.4%
Minerva 540B significantly improves state-of-the-art performance on STEM evaluation datasets.

What Minerva Gets Wrong
Minerva still makes its fair share of mistakes. To better identify areas where the model can be improved, we analyzed a sample of questions the model gets wrong, and found that most mistakes are easily interpretable. About half are calculation mistakes, and the other half are reasoning errors, where the solution steps do not follow a logical chain of thought.

It is also possible for the model to arrive at a correct final answer but with faulty reasoning. We call such cases “false positives”, as they erroneously count toward a model’s overall performance score. In our analysis, we find that the rate of false positives is relatively low (Minerva 62B produces less than 8% false positives on MATH).

Below are a couple of example mistakes the model makes.

Calculation mistake: The model incorrectly cancels the square root on both sides of the equation.
Reasoning mistake: The model computes the number of free throws at the fourth practice, but then uses this number as the final answer for the first practice.

Limitations
Our approach to quantitative reasoning is not grounded in formal mathematics. Minerva parses questions and generates answers using a mix of natural language and LaTeX mathematical expressions, with no explicit underlying mathematical structure. This approach has an important limitation, in that the model’s answers cannot be automatically verified. Even when the final answer is known and can be verified, the model can arrive at a correct final answer using incorrect reasoning steps, which cannot be automatically detected. This limitation is not present in formal methods for theorem proving (e.g., see Coq, Isabelle, HOL, Lean, Metamath, and Mizar). On the other hand, an advantage of the informal approach is that it can be applied to a highly diverse set of problems which may not lend themselves to formalization.

Future Directions
While machine learning models have become impressive tools in many scientific disciplines, they are often narrowly scoped to solve specific tasks. We hope that general models capable of solving quantitative reasoning problems will help push the frontiers of science and education. Models capable of quantitative reasoning have many potential applications, including serving as useful aids for researchers, and enabling new learning opportunities for students. We present Minerva as a small step in this direction. To see more samples from Minerva, such as the one below, please visit the interactive sample explorer!

Solving a problem using calculus and trigonoometry: A question from the MATH dataset asking for the speed of a particle in circular motion. Minerva finds a correct step-by-step solution. In the process, Minerva computes a time derivative and applies a trigonometric identity.

Acknowledgements
Minerva was a collaborative effort that spanned multiple teams in Google Research. We would like to thank our coauthors Aitor Lewkowycz, Ambrose Slone, Anders Andreassen, Behnam Neyshabur, Cem Anil, David Dohan, Henryk Michalewski, Imanol Schlag, Theo Gutman-Solo, Vedant Misra, Vinay Ramasesh, and Yuhuai Wu, as well as our collaborators Erik Zelikman and Yasaman Razeghi. Minerva builds upon the work of many others at Google, and we would like to thank the PaLM team, the T5X team, the Flaxformer team, and the JAX team for their efforts. We thank Tom Small for designing the animation in this post. We would also like to especially thank Vedant Misra for developing the Minerva sample explorer.

Categories
Misc

The Gaming Evolution Will Be Televised: GFN Thursday Levels Up the Living Room Experience on New Samsung TVs and More

Turn the TV on. GeForce NOW is leveling up gaming in the living room. The Samsung Gaming Hub launched today, delivering GeForce NOW natively on 2022 Samsung Smart TVs. Plus, the SHIELD Software Experience Upgrade 9.1 is now rolling out to all NVIDIA SHIELD TVs, delivering new gaming features that improve GeForce NOW. Great living Read article >

The post The Gaming Evolution Will Be Televised: GFN Thursday Levels Up the Living Room Experience on New Samsung TVs and More appeared first on NVIDIA Blog.

Categories
Misc

Is there someone who can I ask help about importing a .tflite in Android studio?

submitted by /u/matticrisp
[visit reddit] [comments]

Categories
Misc

The Metaverse Goes Industrial: Siemens, NVIDIA Extend Partnership to Bring Digital Twins within Easy Reach

Silicon Valley magic met Wednesday with 175 years of industrial technology leadership as Siemens CEO Roland Busch and NVIDIA Founder and CEO Jensen Huang shared their vision for an “industrial metaverse” at the launch of the Siemens Xcelerator business platform in Munich. “When we combine the real and digital worlds we can achieve new levels Read article >

The post The Metaverse Goes Industrial: Siemens, NVIDIA Extend Partnership to Bring Digital Twins within Easy Reach appeared first on NVIDIA Blog.

Categories
Misc

NVIDIA, Partners Show Leading AI Performance and Versatility in MLPerf

NVIDIA and its partners continued to provide the best overall AI training performance and the most submissions across all benchmarks with 90% of all entries coming from the ecosystem, according to MLPerf benchmarks released today. The NVIDIA AI platform covered all eight benchmarks in the MLPerf Training 2.0 round, highlighting its leading versatility. No other Read article >

The post NVIDIA, Partners Show Leading AI Performance and Versatility in MLPerf appeared first on NVIDIA Blog.

Categories
Misc

The Full Stack Optimization Powering NVIDIA MLPerf Training v2.0 Performance

Learn about the full-stack optimizations that enabled the NVIDIA platform to deliver even more performance in MLPerf Training v2.0.

MLPerf benchmarks are developed by a consortium of AI leaders across industry, academia, and research labs, with the aim of providing standardized, fair, and useful measures of deep learning performance. MLPerf training focuses on measuring time to train a range of commonly used neural networks for the following tasks:

  • Natural language processing
  • Speech recognition
  • Recommender systems
  • Biomedical image segmentation
  • Object detection
  • Image classification
  • Reinforcement learning

Lower training times are important to speed time to deployment, minimizing the total cost of ownership and maximizing return on investment.

However, just as important as a platform’s performance is its versatility. The ability to train every model, as well as provide infrastructure fungibility to run all AI workloads from training to inference, is critical to allowing organizations to maximize return on their infrastructure investments.

The NVIDIA platform, with full-stack innovation and a rich developer and application ecosystem, continues to be the only one to submit results on all eight MLPerf Training tests, as well as to submit on all MLPerf Inference and MLPerf high-performance computing (HPC) tests.

In this post, you will learn about the methods that NVIDIA deployed across the entire stack to deliver more performance in MLPerf v2.0.

Full stack improvements

NVIDIA MLPerf v2.0 submissions are based on the proven A100 Tensor Core GPU, the NVIDIA DGX A100 system, as well as the NVIDIA DGX SuperPOD reference architecture. Many partners also submitted results using the A100 Tensor Core GPU.

Through continued innovation across the entire stack, including system software, libraries, and algorithms, NVIDIA has yet again delivered performance improvements compared to prior submissions using the same A100 Tensor Core GPU.

Compared to NVIDIA MLPerf v0.7 submissions, which marked the first A100 Tensor Core GPU submissions, results showed gains of up to 2.1x on a per-chip basis, and 5.7x for max-scale training (Table 1).

Benchmark v2.0 Max-Scale Time to Train (min) (vs. v1.1) (vs. v0.7) v2.0 Per-Accelerator Time to Train (min) (vs. v1.1) (vs. v0.7)
Recommendation (DLRM) 0.59
(1.08x)
(5.66x)
12.78 (A100)
(1.07x)
(2.08x)
Natural Language Processing (BERT) 0.21
(1.10x)
(4.48x)
126.95 (A100)
(1.26x)
(2.69x)
Image Classification (ResNet-50 v1.5) 0.32
(1.09x)
(2.38x)
217.82 (A100)
(1.07x)
(1.46x)
Speech Recognition (RNN-T) 2.15
(1.10x)
(NA*)
230.07 (A100)
(1.19x)
(NA*)
Image Segmentation (3D U-Net) 1.22
(1.13x)
(NA*)
170.23 (A100)
(1.20x)
(NA*)
Object Detection – Lightweight (RetinaNet) 4.25
(NA**)
(NA**)
675.18 (A100)
(NA**)
(NA**)
Object Detection – Heavyweight
(Mask R-CNN)
3.09
(1.05x)
(3.39x)
327.34 (A100)
(1.13x)
(2.01x)
Reinforcement Learning (MiniGo) 16.23
(0.95x)
(1.05x)
2045.37 (A100)
(1.04X)
(1.17x)
Table 1. NVIDIA MLPerf v2.0 training times 

MLPerf v1.1 submission details: 
Per-Accelerator: BERT: 1.1-2066 | DLRM: 1.1-2064 | Mask R-CNN: 1.1-2066 | Resnet50 v1.5: 1.1-2065 | RNN-T: 1.1-2066 | 3D U-Net: 1.1-2065 | MiniGo: 1.1-2067 
Max-Scale: BERT: 1.1-2083 | DLRM: 1.1-2073 | Mask R-CNN: 1.1-2076 | Resnet50 v1.5: 1.1-2082 | SSD: 1.1-2070 | RNN-T: 1.1-2080 | 3D U-Net: 1.1-2077 | MiniGo: 1.1-2081 (*) 

MLPerf v2.0 submission details: 
Per-Accelerator: BERT: 2.0-2070 | DLRM: 2.0-2068   | Mask R-CNN: 2.0-2070 | Resnet50 v1.5: 2.0-2069 | RetinaNet: 2.0-2091 | RNN-T: 2.0-2066 | 3D U-Net: 2.0-2060 | MiniGo: 2.0-2059 
Max-Scale: BERT: 2.0-2106 | DLRM: 2.0-2098 | Mask R-CNN: 2.0-2099 | Resnet50 v1.5: 2.0-2107 | RetinaNet: 2.0-2103 | RNN-T: 2.0-2104 | 3D U-Net: 2.0-2100 | MiniGo: 2.0-2105

Per-Accelerator performance for A100 computed using 8xA100 server time-to-train and multiplying it by 8. 3D U-Net and RNN-T were not part of MLPerf v0.7.  (**)  RetinaNet was not part of either MLPerf v0.7 or v1.1. MLPerf name and logo are trademarks. For more information, see www.mlperf.org.

The following sections highlight some of the work done to achieve these improvements.

BERT

The latest NVIDIA BERT submission took advantage of the following optimizations:  

  • Sequence packing 
  • Fusion of fully connected and GELU layers 

Sequence packing

In previous rounds, the overhead related to padding required to fill up the batch was already optimized by introducing an unpadding optimization. Unpadding, however, results in dynamically sized buffers, as the total number of tokens is not fixed anymore.

This is not an issue when we do not have to use CUDA graphs, such when large-batch sizes are used. However, for small batch sizes, where CUDA graphs are used to reduce CPU overhead, dynamically sized buffers require many separate graphs for each possible size. To take advantage of CUDA graphs efficiently while minimizing the overhead of padding, NVIDIA used the concept of sequence packing this round.

In MLPerf Bidirectional Encoder Representations from Transformers (BERT), a training sample has been restricted to 512 tokens, but it often has fewer tokens than 512. As the training sequences have different lengths, it is possible to fit more than one sequence within a 512-token sample.

Sequence packing requires that the length distribution of the training set sequences be known in advance. The sequences can be merged into packed samples such that none of the merged samples exceed a length of 512 tokens.

NVIDIA used a similar packing algorithm that another submitter used in MLPerf v1.1. Adopting an algorithm from a different submitter was made possible by the high degree of general-purpose programmability of GPUs.

To strike a good balance between implementation complexity and performance, up to three sequences were packed in per sample. This results in each training sample containing a varying number of sequences as a batch of three samples can each contain three to nine sequences.

CUDA graphs require buffer sizes to be fixed across time for each graph. The varying number of total sequences was handled by creating a separate graph for each possible number of sequences within a batch.

For large-scale training, we used a batch size of two per chip. This translates into five to seven separate graphs, which is much less than would be required with the unpadding optimization mentioned at the beginning.

Overall, this technique improved the results for large-scale runs by 10% and 33% for 4096-GPU and 1024-GPU scenarios, respectively.

Fusion of fully connected and GELU layers

BERT uses a Gaussian error linear unit (GELU) activation function that follows a fully connected layer. In prior submissions, the GELU activation function was implemented as a single kernel. This approach required additional memory transactions for both input reads and output writes.

In this round, NVIDIA implemented the fusion of a fully connected layer (a matrix multiply operation) with the GELU activation function. This eliminated the need for a large number of memory read and write operations, yielding a 2-4% increase in overall throughput – larger gains are observed for larger per-chip batch sizes.

In general, it is more efficient to fuse activation math into the end of matrix multiply operations, which means fusing the GELU activation function into different fully connected layers (Figure 1).

An example of matrix multiply operations in which dgrad and wgrad corresponds to data and weight gradient computation for fully connected layers, respectively. The diagram does not show matmul operations with no fusions (such as wgrad_FC1).
Figure 1. Left: Pattern of operations in BERT, middle: Fusion graph in forward pass, right: Fusion graph in backward pass. Each box represents a single kernel.

Deep learning recommendation models

The latest NVIDIA deep learning recommendation models (DLRM) submission again leverages NVIDIA Merlin HugeCTR, an optimized open-source deep neural network training framework for recommender systems.

Kernel fusions

Multilayer perceptrons (MLP) represent a key building block for DLRM. To reduce the number of trips to global memory, fusions of elementwise kernels and general matrix multiply (GEMM) kernels have been widely employed.

The NVIDIA cuBLAS library has recently introduced a new fusion type: GEMM with DReLU (fusing ReLU gradient computation with matrix multiply operations in the backward pass). HugeCTR takes advantage of this new fusion type to enhance the performance of MLPs.

Improved overlap of computation and communication

Increasing GPU utilization is important to provide the highest performance.

In this latest submission, NVIDIA notably improved the overlap of computations and communications in the evaluation of hybrid embeddings to improve GPU utilization. Specifically, the execution of the dense network in iteration i overlapped with the execution of the embedding in iteration i+1 through pipelining, increasing the utilizations of the GPUs.

This overlapping is possible because there are no inter-iteration dependences in the evaluation phase.

Additionally, several of the key kernels in the forward/backward hierarchical all-to-all operations for hybrid embedding were also optimized.

An example of embedding and dense network execution overlapping over three iterations. 
Figure 2. Overlapping execution of embedding and dense network 

ResNet-50

For ResNet-50, we employed the following optimizations to boost performance:  

  • Better max-scale training configuration
  • Faster cuDNN kernels

Better max scale training configuration

When a model is trained on a large scale, if the global batch size is not an integer multiple of the training images in the dataset, the last iteration of the epoch gets added extra data to keep the batch size consistent across iterations. This wasted computation can be saved if the global batch size is close to the integer multiple of the dataset. This is especially important for larger-scale training, where the global batch size is relatively large. 

In this round of MLPerf, we concluded that using 527 nodes with a global batch size of 67,456 significantly reduced wasted computation, resulting in a performance boost of 3.5% compared to NVIDIA’s ResNet-50 submission in MLPerf v1.1.

Faster CuDNN kernels

For the ResNet50 submission, NVIDIA significantly improved the kernels picked up by cuDNN. This includes both better kernels being picked up for the layer sizes and optimized kernel implementation for different tile sizes.

From these optimized kernel samplings, we observed over 4% throughput improvement in the large-scale configurations from MLPerf v1.1.

RetinaNet

The NVIDIA RetinaNet submission takes advantage of several software optimizations, including:  

  • Channel last memory format and automatic mixed precision
  • Using fusion for speed-up
  • Optimizing loss block
  • Asynchronous scoring
  • CUDA Graphs

Channel last memory format and automatic mixed precision

To avoid memory reorganization and to effectively increase peak performance, NVIDIA used the PyTorch channel last memory format (NHWC instead of NCHW) and the PyTorch automatic mixed precision (AMP).

Using fusion for speed-up

For the RetinaNet submission, NVIDIAtook advantage of several fusion opportunities. The cuDNN runtime fusion through the Apex library was used to fuse CONV-bias-ReLU and CONV-bias patterns, and PyTorch NVFuser were used to fuse element-wise operations, such as scale-bias-ReLU and scale-bias-add-ReLU.

The cuDNN runtime fusion Python interface can be found in the Apex repository (import ConvBiasReLU or ConvBias from apex.contrib.conv_bias_relu).

Optimizing loss block

The RetinaNet loss-related calculations are separated into two stages: ground truth data preprocessing and the actual loss calculation.

As the ground truth data preprocessing is not dependent on the model output, part of the ground truth data processing was offloaded to DALI with custom functions, enabling it to be performed asynchronously, improving system resource utilization. The remainder of the preprocessing was reimplemented and then merged into the model graph to avoid jitter.

For the loss calculation, an optimized focal loss implementation was used, which can be found in the Apex library.

Asynchronous scoring

The RetinaNet submission guidelines require that evaluation (inference and scoring) be performed after each training epoch. The scoring time overhead is significant due to the large number of images and bounding boxes in the OpenImages validation dataset, as well as the sequential implementation of the scoring code.

To mitigate the scoring overheads—particularly in large-scale execution, asynchronous scoring was implemented so that the next training epoch masked the previous epoch scoring procedure.

Diagram shows the execution of asynchronous COCO scoring as part of MLPerf RetinaNet evaluation phase
Figure 3. Execution of asynchronous COCO scoring in evaluation

CUDA Graphs

CUDA Graphs were used extensively in the NVIDIA RetinaNet submission. The entire model and portions of the ground truth preprocessing were graphed which required that they be reimplemented to fit CUDA graph constraints.

The model’s forward and backward passes were graph captured, as well as portions of the ground truth preprocessing. The latter required code adaptation to fit CUDA graph constraints.

For more information, see Accelerating PyTorch with CUDA Graphs.

Mask R-CNN

The NVIDIA Mask R-CNN submissions utilized several techniques to improve performance:  

  • Bottleneck block optimizations
  • RPN head fusion
  • Evaluation
  • Top-K

Bottleneck block optimizations

The resnet backbone is built as a stack of bottleneck blocks, each composed of three sequential convolutions. Each convolution is followed by a batch-norm and a ReLu. The batch-norm modules have four parameters, and some math is required to compute a couple of intermediate terms in the forward method.

As the batch-norms are frozen, the parameters never change, meaning that the intermediate terms do not change either. To save time, these intermediate terms were computed just one time.

Backpropagation of ReLu involves creating and applying a mask. In earlier versions of the code, this mask was stored with half (FP16) precision. In this round, the DReLU mask is represented as a Boolean and not FP16 to reduce memory transactions.

During back propagation, data gradients and weight gradients were computed for each of the three convolution layers. NVIDIA found empirically that while the data gradient GPU kernels were launched with a sufficient number of CTAs to fully use the GPUs, the weight gradient kernels were launched with far fewer CTAs.

One optimization that was implemented was to launch the data gradient kernels first, and then launch all three weight gradient kernels on separate streams so that they ran concurrently. This reduced total computation time for weight gradients.

These optimizations are available for PyTorch users in the bottleneck block module in Apex.

RPN head fusion

A new Apex module, which fuses convolution, bias, and ReLu, was implemented, as discussed in the RetinaNet section. This module was in MaskR-CNN as well to fuse forward propagation of some of the layers in RPN head block.

Evaluation

Evaluation, on average, takes almost as much time as training. Evaluation is done asynchronously on dedicated nodes, but the results are shared with the training nodes through a blocking broadcast.

The training nodes wait for a certain number of steps before they start waiting for the evaluation broadcast to minimize any evaluation result wait time. The learning rate curve has two inflection points in it, and it is extremely unlikely that the model will have converged before passing the last inflection point. That’s why you should wait for as long as possible to check for evaluation results, until training has passed the last learning rate curve inflection point.

Top-K

In earlier versions of PyTorch, the number of CTAs launched by the top-k kernel was proportional to the per-GPU batch size. This yielded poor performance when the batch size equaled 1, the batch size that is always used for NVIDIA max-scale runs.

This issue was addressed in previous rounds with a two-stage top-k method, which was implemented in Python, but this solution did not generalize well. Work on a more general solution was already underway.

In this round, NVIDIA worked with the PyTorch team to ensure that the new top-k implementation that yielded far better performance for a batch size of 1 made it into PyTorch. When this was complete, the prior two-stage top-k implementation was replaced with the new PyTorch module.

3D U-Net

3D U-Net has multiple large layers with an input channel count of 32. For wgrad kernels, using a kernel with the default 64x256x64 meant significant tile size quantization loss.

Thanks to the introduction of the new 32x256x32 wgrad kernels in cuDNN, this tile size quantization loss was saved. This resulted in a speedup of over 5% on a single node in MLPerf v2.0 over MLPerf v1.1.

RNN-T

The preprocessing step of Recurrent Neural Network Transducer (RNN-T) is relatively intensive. Thanks to DALI, most of the preprocessing overhead can be pipelined and hidden under the main training loop.

However, because the size of the input data might vary, there was a need for relocating the internal memory buffers after the initial iteration, increasing the length of the warm-up phase.

DALI has recently switched to a memory-pool based allocator, where the pool is managed using the cuMem API. This significantly reduces the overhead of allocating new buffers, yielding a much faster warm-up process in training.

Conclusion

Thanks to optimizations across the stack, the NVIDIA platform was yet again able to boost performance in MLPerf Training v2.0 using the proven NVIDIA A100 Tensor Core GPU and NVIDIA DGX A100 platforms.

NVIDIA continues to be the only platform to submit results in the MLPerf benchmarking suite, including MLPerf Training, MLPerf Inference, and MLPerf HPC. This showcases the performance and versatility of the entire platform, which is crucial as modern AI becomes pervasive across every computing domain.

In addition to providing the software used for NVIDIA MLPerf submissions in the MLPerf repository, dozens of additional models were also made and optimized for NVIDIA GPUs, available on the NGC hub.

The NVIDIA platform is also ubiquitous, providing customers with the choice of where to run models. NVIDIA A100 is available from all major server makers and cloud service providers, allowing you to deploy on-premises, in the cloud, in a hybrid environment, or at the edge.

Categories
Misc

Siemens and NVIDIA to Enable Industrial Metaverse

Siemens, a leader in industrial automation and software, infrastructure, building technology and transportation, and NVIDIA, a pioneer in accelerated graphics and artificial intelligence (AI), today announced an expansion of their partnership to enable the industrial metaverse and increase use of AI-driven digital twin technology that will help bring industrial automation to a new level.

Categories
Misc

Improving Enterprise IT Fraud Prevention

This post discusses infrastructure factors to consider, such as performance, hardware, and types of AI software for implementing a fraud prevention strategy.

Any business or industry, from retail and healthcare to financial services, is subject to fraud. The cost of fraud can be staggering. Every $1 of fraud loss costs financial firms about $4 to mitigate. Online sellers will lose $130B to online payment fraud between 2018 and 2023.

By using AI and big data analytics, enterprises can efficiently prevent fraud attempts in real time.

This post discusses the infrastructure factors to consider, such as performance, hardware, and types of AI software for implementing a fraud prevention strategy.

Pre– and post-transaction fraud detection

Before discussing fraud detection, let’s be clear on the difference between prevention and detection. Fraud prevention describes the overall effort to manage and eliminate fraud. Fraud detection is simply the ability to identify fraudulent activity. 

There are two approaches to fraud detection, both necessary for a comprehensive fraud prevention strategy.

  • Pre-transaction detection: Detecting and blocking attempted fraud transactions before transaction completion. When anomalous data or behavior is detected before a transaction, the transaction is blocked.
  • Post-transaction detection: A fraudulent transaction is identified after transaction completion based on data analysis or post-transaction inputs. This is followed by damage mitigation.

The ideal approach is to detect and block attempted fraud before its occurrence. When fraud is detected after a transaction, the only recourse is to assess the damage, notify relevant parties, and work to recover from the fraud damage.

Although fraud can never fully be eliminated, both pre– and post-transaction fraud efforts are important in developing a fraud risk management plan.

Best enterprise IT practices for developing an effective anti-fraud solution

If fraud prevention was as simple as instant breakfast cereal, you’d just add hot water and stir. A standard server and software is all that’s needed for effective fraud prevention. Correct? Not exactly.

Fraud prevention software is obviously essential, but choosing just any hardware and software combination does not ensure success. In spite of widespread fraud prevention “solutions” across enterprises, fraud continues to increase, causing financial damage in the process.

Enterprise IT must ensure that multiple infrastructure elements are in place:

  • AI-driven software: There is a trend toward AI-driven fraud solutions because traditional static intelligence is not as effective as the dynamic intelligence of AI. To prevent sophisticated fraud attempts, software must learn. Thus, AI must be at the core.
  • Accelerated performance: Real-time AI-driven fraud detection requires the highest performance possible. Delays can impact customer experience. With performance, more fraud factors can be evaluated in real time, resulting in more accurate fraud detection.
  • Availability and scale: A highly available scale-out architecture is required to support 24-7 data ingest and prevention.

Without these three components, you can expect less efficiency, which can result in greater fraud loss for businesses and customers.

AI-driven software

AI-driven software is already pervasive in the enterprise. Through AI-trained fraud prevention models, the accuracy of detecting real fraud can be evaluated and tuned based on model training iteration. After training, the prevention solution runs as an inferencing application to evaluate and block potentially fraudulent transactions. Data from the application is then fed back to the model for re-training and greater accuracy and efficiency. The continuously retrained model is then used to update the production application and so on.

As machine learning (ML) and deep learning (DL) are increasingly applied to growing datasets, Apache Spark has become popular for data preprocessing and feature engineering as raw data is prepared for training. It is also used as a data execution engine for analytics workloads. GPUs parallelize and accelerate data processing queries in the same way they accelerate deep-learning and AI workloads. RAPIDS is commonly used to accelerate Spark as well as ML/DL frameworks within a GPU-accelerated infrastructure.

In addition, AI frameworks like NVIDIA Morpheus can run in an unsupervised manner to flag anomalous activity and enhance fraud prevention efforts. AI-based fraud prevention is dynamic and can automatically adapt to threats.

Diagram shows NVIDIA Morpheus-driven AI framework  that includes RAPIDS, cyber log accelerators, Triton Inference Server, and TensorRT.
Figure 1. Example of an NVIDIA Morpheus-driven framework

Accelerated performance

Yes, there is a need for speed, especially for pre-transaction fraud detection.

Pre-transaction detection

Without hardware acceleration, customers executing transactions can be inconvenienced with unacceptable delays as fraud prevention software analyzes each transaction. Slow processing impacts customer satisfaction and merchants realize less revenue. Standard CPUs were sufficient for legacy fraud prevention, but no longer.

Modern AI-based solutions powered by enterprise-class GPUs can be an order of magnitude more effective in speed and accuracy. High-performance GPU acceleration can enable the evaluation of more risk factors in a given time frame. Alternatively, the same number of risk factors can be evaluated in less time.

Post-transaction detection

Post-transaction fraud detection is not bound by real-time constraints. Still, GPU acceleration can provide benefits. Faster processing power allows for more data to be evaluated within a given period of time.

As with pre-transaction results, post-transaction results can be used to update pre– and post-processing models to improve future results (inferencing).

Other performance considerations

IT departments may mistakenly evaluate server performance requirements solely based on transaction speed. However, high performance is also required for initial model training and retraining based on incoming feedback data from the inference application.

Without GPU acceleration, initial training can take hours or days, and retraining can be unacceptably long. Time-to-results is not just about GPU clock rate. Some GPUs are not certified as enterprise-class and lack substantial GPU memory or a sufficient number of cores to deliver fast training results.

Yes, training of large fraud prevention models can be executed in the cloud with both performance and scale. Unfortunately, cloud processing cycles can be expensive after multiple training iterations.

Fortunately, model training can be cost-effectively executed locally in an iterative manner on enterprise GPU-accelerated workstations using a representative subset of data or with lower-resolution accuracy parameters. This makes the cost of preliminary model training essentially free after the capital expense of data science workstations or GPU-enabled servers.

After preliminary training, full-scale training in the cloud or enterprise server or server cluster can be executed on a large data set with greater efficiency.

Availability and scale

Fraudsters never sleep, so enterprise fraud prevention also cannot rest. Transactional apps run non-stop, so fraud prevention software must do the same. Enterprise IT infrastructure must deliver resiliency and availability for the prevention solution. As I’ve noted, performance is important, but when it’s not always available, it negates any performance benefits.

As I discussed earlier, fraud and associated damage are increasing each year. Prevention solutions must seamlessly scale out to accommodate this.

Availability and scalability requirements cannot be limited to servers. For example, a network may meet the specs for required throughput and latency, but the reality is that burst network traffic may cause sufficient network congestion such that fraud detection is either skipped, unacceptably delayed, or timed out. For this reason, network robustness and redundancy can’t be ignored when building an advanced anti-fraud solution.

Building AI-friendly infrastructure for fraud solutions

Will your AI-driven solution receive the performance, availability, and scale that is needed for success over time? Assuming the right AI-based fraud prevention software, IT must also ensure the right infrastructure. As I discussed earlier, this means infrastructure for accelerated performance, non-stop operation, and the flexibility to scale out seamlessly the infrastructure investment as fraud prevention data and workloads increase over time.

It is also challenging to deliver enterprise-class IT infrastructure that supports both fraud prevention and AI-driven enterprise solutions. The right mix of products addresses the performance, availability, and scale-out requirements for enterprise fraud prevention while supporting other AI frameworks and tools. This can range from mobile and desktop workstations for model development to servers and software for data center inference and training at scale.

For more information about fraud detection and prevention for demanding enterprise AI environments, see Fraud Detection – Top Resources from GTC 21.

Categories
Misc

NVIDIA Studio Driver Elevates Creative Workflows in Blender 3.2, BorisFX Sapphire and Topaz Denoise AI

The June NVIDIA Studio Driver is available for download today, optimizing the latest creative app updates, all with the stability and reliability that users count on. Creators with NVIDIA RTX GPUs will benefit from faster performance and new features within Blender version 3.2, BorisFX Sapphire release 2022.5 and Topaz Denoise AI 3.7.0.

The post NVIDIA Studio Driver Elevates Creative Workflows in Blender 3.2, BorisFX Sapphire and Topaz Denoise AI appeared first on NVIDIA Blog.