DataBloomSensor AI solutions specialist SenSen has turned to the NVIDIA Jetson edge AI platform to help regulators track heavy vehicles moving across Australia. Australia’s National Heavy Vehicle Regulator, or NHVR, has a big job — ensuring the safety of truck drivers across some of the world’s most sparsely populated regions. They’re now harnessing AI to Read article >
The post Keep On Trucking: SenSen Harnesses Drones, NVIDIA Jetson, Metropolis to Inspect Trucks appeared first on NVIDIA Blog.
Join us on October 24 for a deep dive into MONAI, the essential framework for AI workflows in healthcare—including use cases, building blocks, and more.
Join us on October 24 for a deep dive into MONAI, the essential framework for AI workflows in healthcare—including use cases, building blocks, and more.
Explore deep learning with hands-on exercises in computer vision and NLP in this online instructor-led workshop.
Explore deep learning with hands-on exercises in computer vision and NLP in this online instructor-led workshop.
Speech recognition technology is growing in popularity for voice assistants and robotics, for solving real-world problems through assisted healthcare or…
Speech recognition technology is growing in popularity for voice assistants and robotics, for solving real-world problems through assisted healthcare or education, and more. This is helping democratize access to speech AI worldwide. As labeled datasets for unique, emerging languages become more widely available, developers can build AI applications readily, accurately, and affordably to enhance technology developments and experiences for their native regions.
Kinyarwanda is the native language of 9.8 million people in Rwanda, Uganda, DR Congo, and Tanzania with over 20 million total speakers across the globe.
In April 2022, Mozilla Common Voice (MCV), a crowdsourced project aimed at making voice recognition open and accessible to everyone, made a significant contribution to building the Kinyarwanda dataset, as detailed in the article, Lessons from Building for Kinyarwanda on Common Voice. It is a 57 GB dataset with 2,000+ hours of audio, making it the largest dataset on the MCV platform.
To bring the value of the effort and dataset to developers, an automatic speech recognition (ASR) model was trained on this dataset that achieved state-of-the-art performance on the published checkpoints.
This post provides an overview of the training process using NeMo ASR toolkit. It briefly covers challenges with the dataset, converting characters to longer units using byte-pair encoding, and the training process for improved model performance. Developers can refer to the step-by-step tutorial on GitHub for the reference code and details.
MCV has the largest publicly available multi-language dataset. You can download language-specific datasets from the Mozilla Common Voice Hub.
In the Kinyarwanda dataset used for the model, there are 1,404,853 sentences that are pre-split into train/dev/test data. Each entry in the dataset consists of a unique MP3 file and corresponding information such as name of the file, transcription, and meta information in TSV format.
NeMo ASR requires data that includes a set of utterances in individual audio files plus a manifest that describes the dataset, with information about one utterance per line.
Once the dataset is downloaded, in the training split, TSV files are converted to JSON manifests and MP3 files are converted to WAV files, which are recommended formats for NeMo toolkit. The same steps are then repeated for test and dev data separately.
The manifest format is provided below:
{"audio_filepath": "/path/to/audio.wav", "text": "the transcription of the utterance", "duration": 23.147}Before training the model, the data requires preprocessing to reduce ambiguity and inconsistencies and make the data easy to interpret. The preprocessing steps for this model are:
(combined Latin letters, space, and apostrophe, for example)
Because 99% of the dataset has an audio duration of 11 seconds or shorter, it is suggested to restrict the maximum audio duration to 11 seconds during preprocessing for faster training.
The final Kinyarwanda transcript consists of sentences with Latin letters, spaces, and apostrophes after preprocessing.
It is possible to train character-based ASR models but they will regard each letter as a separate token, taking more time to generate the output. Using longer units improves both quality and speed.
This process involves a tokenization algorithm called byte-pair encoding that splits words into subtokens and marks the beginning of the word with a special symbol so it’s easy to restore the original words.
To make the process easier, NeMo toolkit supports on-the-fly subword tokenization by passing the tokenizer through the model config so there is no need to modify transcripts. This does not affect the model performance and potentially helps to adapt to other domains without retraining the tokenizer.
Visit NVIDIA/NeMo on GitHub for a detailed description and tutorial on subword tokenization for NeMo ASR.
Two approaches lead to trained model. The first approach involves training the model from scratch using two model architectures: Conformer-CTC and Conformer-Transducer. The second approach involves
fine-tuning the Kinyarwanda Conformer-Transducer model from different pretrained checkpoints.
To train a Conformer-CTC model, use speech_to_text_ctc_bpe.py with the default config conformer_ctc_bpe.yaml. To train a Conformer-Transducer model, use speech_to_text_rnnt_bpe.py with the default config conformer_transducer_bpe.yaml.
For fine-tuning, use the pretrained STT_EN_Conformer_Transducer model for a checkpoint that is not self-supervised. Use the SSL_EN_Conformer_Large for a self-supervised checkpoint from NVIDIA GPU Cloud.
You can find more details about the training process in the step-by-step tutorial on GitHub.
The reference code for Self-supervised Checkpoint Initialization (SSL_EN_Conformer_Large) is provided below.
import nemo.collections.asr as nemo_asr
ssl_model = nemo_asr.models.ssl_models.SpeechEncDecSelfSupervisedModel.from_pretrained(model_name='ssl_en_conformer_large')
# define fine-tune model
asr_model = nemo_asr.models.EncDecCTCModelBPE(cfg=cfg.model, trainer=trainer)
# load ssl checkpoint
asr_model.load_state_dict(ssl_model.state_dict(), strict=False)
del ssl_model
Figure 1 shows a comparison of training dynamics. The fine-tuning approach is quick and easy for training, and also leads to faster convergence and better quality.
While building a model, the goal is to minimize the Word Error Rate (WER) while transcribing the speech input. In simple words, Word Error Rate is the number of errors divided by the total number of words. It is often used to test the performance of a model but should not be the only standard, as out-of-scope variables like noise, echo, and accents can have a substantial impact on speech recognition.
Character Error Rate (CER) is also considered. CER gives the percentage of characters that were incorrectly predicted. Our models have the lowest percentage of WER and CER in the Kinyarwanda ASR models (Table 1).
| Model | WER % | CER % |
| Conformer-CTC-Large | 18.73 | 5.75 |
| Conformer-Transducer-Large | 16.19 | 5.7 |
We have built two high-quality Kinyarwanda checkpoints from scratch with the NeMo toolkit. The Conformer-Transducer checkpoint has better quality but the Conformer-CTC is 4x faster at inference, so they are both potentially useful based on the need.
The high performance of the pretrained model is another step towards new developments in the speech AI community. The state-of-the-art model can be improved further by fine-tuning it with more data that has more dialects, accents, and rare words and is a true representation of how people speak their native languages. NVIDIA NeMo pretrained models are open source and meet the goal of democratization and inclusivity across the globe.
Explore the MVC initiative to access or provide voice data for your language. For more information on models, see the following resources:
Join experts from Google, Meta, NVIDIA, and more at the first annual NVIDIA Speech AI Summit. Register now.
GeForce NOW expands touch control support to 13 more games this GFN Thursday. That means it’s easier than ever to take PC gaming on the go using mobile devices and tablets. The new “Mobile Touch Controls” row in the GeForce NOW app is the easiest way for members to find which games put the action Read article >
The post Get in Touch With New Mobile Gaming Controls on GeForce NOW appeared first on NVIDIA Blog.
At ROSCon 2022, NVIDIA announced the newest Isaac ROS software release, Developer Preview (DP) 2. This release includes new cloud– and edge-to-robot task…
At ROSCon 2022, NVIDIA announced the newest Isaac ROS software release, Developer Preview (DP) 2. This release includes new cloud– and edge-to-robot task management and monitoring software for autonomous mobile robot (AMR) fleets, as well as additional features for ROS 2 developers.
NVIDIA Isaac ROS consists of individual packages (GEMs) and complete pipelines (NITROS) for hardware-accelerated performance. In addition to performance improvements, the new release adds the following functionality:
Mission Dispatch and Client provide a standard, open-source way to assign and track tasks between a fleet management system and ROS 2 robots. Dispatch and Client communicate using VDA5050, an open standard for communications designed specifically for robot fleets. Messages are transmitted wirelessly over MQTT, a lightweight messaging protocol for Internet of Things (IoT) applications.
Mission Dispatch is a containerized micro-service available for download from NGC, or as source code on the NVIDIA Isaac GitHub repo, and can be integrated into fleet management systems. Mission Dispatch has been verified to interoperate with other open-source ROS 2 clients like the recently announced VDA5050 Connector developed by OTTO Motors and InOrbit.
Mission Client, which is compatible with ROS 2 Humble, is available as a package in the NVIDIA Isaac ROS GitHub repo and preintegrated with the Nav2 navigation stack to assign and track navigation and other tasks on the robot.
“As mobile robot deployment in the real world accelerates, interoperability is becoming increasingly critical,” said Ryan Gariepy, CTO at OTTO Motors. “Bridging VDA5050 with ROS2 as an open-source community will promote innovation in fleet management solutions while allowing robot makers to focus on differentiation.”
NVIDIA Isaac ROS continues to deliver hardware-accelerated performance for the ROS 2 developer community for AI perception, image processing, and navigation. Autonomous robots require advanced AI and computer vision capabilities. Isaac ROS represents our commitment to making it easier for the robotics community to adopt these cutting-edge technologies.
For more information about the latest performance numbers for key Isaac ROS packages, see Isaac ROS Performance Summary.
To provide advanced technical training and access to NVIDIA Isaac ROS experts, NVIDIA is announcing a new series of webinars focused on ROS 2 developers. These sessions are free and feature Q&A periods with the technical experts developing accelerated modules for ROS 2.
The first three webinar topics:
Register for the November 14 webinar and check back soon, as more webinars will be added to the series.
If you are attending ROSCon in Kyoto, Japan, be sure to attend the technical session gz-omni: Bridging Gazebo with Isaac Sim (livestream) on October 20, 2022 at 2:10PM JST. Visit NVIDIA at booth #22 to see a live demonstration of NVIDIA Isaac ROS in action running on the NVIDIA Jetson AGX Orin Developer Kit.
To get started today with NVIDIA Isaac ROS, review the examples summarized in the /NVIDIA-ISAAC-ROS GitHub repo.
There are some 1.8 billion Muslims, but only 16% or so of them speak Arabic, the language of the Quran. This is in part due to the fact that many Muslims struggle to find qualified instructors to give them feedback on their Quran recitation. Enter today’s guest and his company Tarteel, a member of the Read article >
The post How Tarteel Uses AI to Help Arabic Learners Perfect Their Pronunciation appeared first on NVIDIA Blog.
Path tracing is going real-time, unleashing interactive, photorealistic 3D environments filled with dynamic light and shadow, reflections, and refractions.
Path tracing is going real-time, unleashing interactive, photorealistic 3D environments filled with dynamic light and shadow, reflections, and refractions.
If you’ve used a chatbot, predictive text to finish a thought in an email, or pressed “0” to speak to an operator, you’ve come across natural language…
If you’ve used a chatbot, predictive text to finish a thought in an email, or pressed “0” to speak to an operator, you’ve come across natural language processing (NLP). As more enterprises adopt NLP, the sub-field is developing beyond those popular use cases of machine-human communication to machines interpreting both human and non-human language. This creates an exciting opportunity for organizations to stay ahead of evolving cybersecurity threats.
This post was originally published on CIO.com
NLP combines linguistics, computer science, and AI to support machine learning of human language. Human language is astonishingly complex. Relying on structured rules leaves machines with an incomplete understanding of it.
NLP enables machines to contextualize and learn instead of relying on rigid encoding so that they can adapt to different dialects, new expressions, or questions that the programmers never anticipated.
NLP research has driven the evolution of AI tech, like neural networks that are instrumental to machine learning across various fields and use cases. NLP has been primarily leveraged across machine-to-human communication to simplify interactions for enterprises and consumers.
NLP was designed to enable machines to learn to communicate like humans, with humans. Many services that we use today leverage machine communications either to each other or in translation to become intelligible by humans. Cybersecurity is the perfect example of such a field where IT analysts can feel like they speak to more machines than people.
NLP can be leveraged in cybersecurity workflows to assist in breach protection, identification, and scale and scope analysis.
In the short term, NLP can be easily leveraged to enhance and simplify breach protection from phishing attempts.
In the context of phishing, NLP can be leveraged to understand bot or spam behavior in email text sent by a machine posing as a human. It can also be used to understand the internal structure of the email itself to identify patterns of spammers and the types of messages they send.
This example is the first extension of NLP, originally designed to understand just human language and now being applied to understand the combination of human language mixed with machine-level headers.
In the medium term, NLP can be leveraged to parse logs, a cyBERT use case.
In the current rules-based system, the mechanisms and systems required to parse raw logs and make them ready for analysts are brittle and need significant development and maintenance resources.
Using NLP, parsing of raw logs becomes more flexible and less prone to breaking when changes occur to the log generators and sensors.
Going further, the neural networks used for parsing can generalize beyond the logs they were exposed to during training, creating methods to transform raw data into rich content ready for an analyst without the need to write explicit rules for these new or changed log types.
As a result, NLP models are more accurate at parsing logs than traditional rules while being more flexible and fault-tolerant.
In the longer term, entirely synthetic languages can be created that represent machine-to-machine and human-to-machine communications.
If two machines can create an entirely new language, that language can then be analyzed using NLP techniques to identify errors in grammar, syntax, and composition. All these can be interpreted as anomalies and contextualized for analysts.
This new development can help identify known issues or attacks when they occur, and can also identify completely unknown misconfigurations and attacks, which helps analysts be more efficient and effective.
The phishing protection, log parsing, and synthetic language applications are just the beginning for NLP. To learn more about AI and cybersecurity, see Learn About the Latest Developments with AI-Powered Cybersecurity, one of many on-demand sessions from NVIDIA GTC.
Single-cell measurement technologies have advanced rapidly, revolutionizing the life sciences. We have scaled from measuring dozens to millions of cells and…
Single-cell measurement technologies have advanced rapidly, revolutionizing the life sciences. We have scaled from measuring dozens to millions of cells and from one modality to multiple high dimensional modalities. The vast amounts of information at the level of individual cells present a great opportunity to train machine learning models to help us better understand the intrinsic link of cell modalities, which could be transformative for synthetic biology and drug target discovery.
This post introduces modality prediction and explains how we accelerated the winning solution of the NeurIPS Single-Cell Multi-Modality Prediction Challenge by drop-in replacing the CPU-based TSVD and kernel ridge regression (KRR), implemented in scikit-learn, with NVIDIA GPU-based RAPIDS cuML implementations.
Using cuML and changing only six lines of code, we accelerated the scikit-learn–based winning solution, reducing the training time from 69 minutes to 40 seconds: a 103.5x speedup! Even when compared to sophisticated deep learning models developed in PyTorch, we observed that the cuML solution is both faster and more accurate for this prediction challenge.
Thanks to single-cell technology, we can measure multiple modalities within the same single cell such as DNA accessibility (ATAC), mRNA gene expression (GEX), and protein abundance (ADT). Figure 1 shows that these modalities are intrinsically linked. Only accessible DNA can produce mRNA, which in turn is used as a template to produce protein.
The problem of modality prediction arises naturally where it is desirable to predict one modality from another. In the 2021 NeurIPS challenge, we were asked to predict the flow of information from ATAC to GEX and from GEX to ADT.
If a machine learning model can make good predictions, it must have learned intricate states of the cell and it could provide a deeper insight into cellular biology. Extending our understanding of these regulatory processes is also transformative for drug target discovery.
The modality prediction is a multi-output regression problem, and it presents unique challenges:
In this post, we focus on the task of GEX to ADT predictions to demonstrate the efficiency of a single-GPU solution. Our methods can be extended to other single-cell modality prediction tasks with larger data size and higher cardinality using multi-node multi-GPU architectures.
As our baseline, we used the first-place solution of the NeurIPS Modality Prediction Challenge “GEX to ADT” from Kaiwen Deng of University of Michigan. The workflow of the core model is shown in Figure 2. The training data includes both GEX and ADT information while the test data only has GEX information.
The task is to predict the ADT of the test data given its GEX. To address the sparsity and redundancy of the data, we applied truncated singular value decomposition (TSVD) to reduce the dimension of both GEX and ADT.
In particular, two TSVD models fit GEX and ADT separately:
In Deng’s solution, dimensionality is reduced aggressively from 13953 to 300 for GEX and from 134 to 70 for ADT.
The number of principal components 300 and 70 are hyperparameters of the model, which are obtained through cross-validation and tuning. The reduced version of GEX and ADT of training data are then fed into KRR with the RBF kernel. Matching Deng’s approach, at inference time, we used the trained KRR model to perform the following tasks:
Generally, TSVD is the most popular choice to perform dimension reduction for sparse data, typically used during feature engineering. In this case, TSVD is used to reduce the dimension of both the features (GEX) and the targets (ADT). Dimension reduction of the targets makes it much easier for the downstream multi-output regression model because the TSVD outputs are more independent across the columns.
KRR is chosen as the multi-output regression model. Compared to SVM, KRR computes all the columns of the output concurrently while SVM predicts one column at a time so KRR can learn the nonlinearity like SVM but be much faster.
cuML is one of the RAPIDS libraries. It contains a suite of GPU-accelerated machine learning algorithms that provide many highly optimized models, including both TSVD and KRR. You can quickly adapt the baseline model from a scikit-learn implementation to a cuML implementation.
In the following code example, we only needed to change six lines of code and three of them are imports. For simplicity, many preprocessing and utility codes are omitted.
Baseline sklearn implementation:
from sklearn.decomposition import TruncatedSVD
from sklearn.gaussian_process.kernels import RBF
from sklearn.kernel_ridge import KernelRidge
tsvd_gex = TruncatedSVD(n_components=300)
tsvd_adt = TruncatedSVD(n_components=70)
gex_train_test = tsvd_gex.fit_transform(gex_train_test)
gex_train, gex_test = split(get_train_test)
adt_train = tsvd_adt.fit_transform(adt_train)
adt_comp = tsvd_adt.components_
y_pred = 0
for seed in seeds:
gex_tr,_,adt_tr,_=train_test_split(gex_train,
adt_train,
train_size=0.5,
random_state=seed)
kernel = RBF(length_scale = scale)
krr = KernelRidge(alpha=alpha, kernel=kernel)
krr.fit(gex_tr, adt_tr)
y_pred += (krr.predict(gex_test) @ adt_comp)
y_pred /= len(seeds)
RAPIDS cuML implementation:
from cuml.decomposition import TruncatedSVD
from cuml.kernel_ridge import KernelRidge
import gc
tsvd_gex = TruncatedSVD(n_components=300)
tsvd_adt = TruncatedSVD(n_components=70)
gex_train_test = tsvd_gex.fit_transform(gex_train_test)
gex_train, gex_test = split(get_train_test)
adt_train = tsvd_adt.fit_transform(adt_train)
adt_comp = tsvd_adt.components_.to_output('cupy')
y_pred = 0
for seed in seeds:
gex_tr,_,adt_tr,_=train_test_split(gex_train,
adt_train,
train_size=0.5,
random_state=seed)
krr = KernelRidge(alpha=alpha,kernel='rbf')
krr.fit(gex_tr, adt_tr)
gc.collect()
y_pred += (krr.predict(gex_test) @ adt_comp)
y_pred /= len(seeds)
The syntax of cuML kernels is slightly different from scikit-learn. Instead of creating a standalone kernel object, we specified the kernel type in the KernelRidge’s constructor. This is because the Gaussian process is not supported by cuML yet.
Another difference is that explicit garbage collection is needed for the current version cuML implementations. Some form of reference cycles are created in this particular loop and objects are not freed automatically without garbage collection. For more information, see the complete notebooks in the /daxiongshu/rapids_nips_blog GitHub repo.
We compared the cuML implementation of TSVD+KRR against the CPU baseline and other top solutions in the challenge. The GPU solutions run on a single V100 GPU and the CPU solutions run on dual 20-core Intel Xeon CPUs. The metric for the competition is root mean square error (RMSE).
We found that the cuML implementation of TSVD+KRR is 103x faster than the CPU baseline with a slight degradation of the score due to the randomness in the pipeline. However, the score is still better than any other models in the competition.
We also compared our solution with two deep learning models:
Both deep learning models are implemented in PyTorch and run on a single V100 GPU. Both deep learning models have many layers with millions of parameters to train and hence are prone to overfitting for this dataset. In comparison, TSVD+KRR only has to train less than 30K parameters. Figure 4 shows that the cuML TSVD+KRR model is both faster and more accurate than the deep learning models, thanks to its simplicity.
Figure 5 shows a detailed speedup analysis, where we present timings for the two stages of the algorithm: TSVD and KRR. cuML TSVD and KRR are 15x and 103x faster than the CPU baseline, respectively.
Figure 5. Run time comparison
Due to its lightning speed and user-friendly API, RAPIDS cuML is incredibly useful for accelerating the analysis of the single-cell data. With a few minor code changes, you can boost your existing scikit-learn workflows.
In addition, when dealing with single-cell modality prediction, we recommend starting with cuML TSVD to reduce the dimension of data and KRR for the downstream tasks to achieve the best speedup.
Try out this RAPIDS cuML implementation with the code on the /daxiongshu/rapids_nips_blog GitHub repo.