Categories
Offsites

Private Ads Prediction with DP-SGD

Ad technology providers widely use machine learning (ML) models to predict and present users with the most relevant ads, and to measure the effectiveness of those ads. With increasing focus on online privacy, there’s an opportunity to identify ML algorithms that have better privacy-utility trade-offs. Differential privacy (DP) has emerged as a popular framework for developing ML algorithms responsibly with provable privacy guarantees. It has been extensively studied in the privacy literature, deployed in industrial applications and employed by the U.S. Census. Intuitively, the DP framework enables ML models to learn population-wide properties, while protecting user-level information.

When training ML models, algorithms take a dataset as their input and produce a trained model as their output. Stochastic gradient descent (SGD) is a commonly used non-private training algorithm that computes the average gradient from a random subset of examples (called a mini-batch), and uses it to indicate the direction towards which the model should move to fit that mini-batch. The most widely used DP training algorithm in deep learning is an extension of SGD called DP stochastic gradient descent (DP-SGD).

DP-SGD includes two additional steps: 1) before averaging, the gradient of each example is norm-clipped if the L2 norm of the gradient exceeds a predefined threshold; and 2) Gaussian noise is added to the average gradient before updating the model. DP-SGD can be adapted to any existing deep learning pipeline with minimal changes by replacing the optimizer, such as SGD or Adam, with their DP variants. However, applying DP-SGD in practice could lead to a significant loss of model utility (i.e., accuracy) with large computational overheads. As a result, various research attempts to apply DP-SGD training on more practical, large-scale deep learning problems. Recent studies have also shown promising DP training results on computer vision and natural language processing problems.

In “Private Ad Modeling with DP-SGD”, we present a systematic study of DP-SGD training on ads modeling problems, which pose unique challenges compared to vision and language tasks. Ads datasets often have a high imbalance between data classes, and consist of categorical features with large numbers of unique values, leading to models that have large embedding layers and highly sparse gradient updates. With this study, we demonstrate that DP-SGD allows ad prediction models to be trained privately with a much smaller utility gap than previously expected, even in the high privacy regime. Moreover, we demonstrate that with proper implementation, the computation and memory overhead of DP-SGD training can be significantly reduced.

Evaluation

We evaluate private training using three ads prediction tasks: (1) predicting the click-through rate (pCTR) for an ad, (2) predicting the conversion rate (pCVR) for an ad after a click, and 3) predicting the expected number of conversions (pConvs) after an ad click. For pCTR, we use the Criteo dataset, which is a widely used public benchmark for pCTR models. We evaluate pCVR and pConvs using internal Google datasets. pCTR and pCVR are binary classification problems trained with the binary cross entropy loss and we report the test AUC loss (i.e., 1 – AUC). pConvs is a regression problem trained with Poisson log loss (PLL) and we report the test PLL.

For each task, we evaluate the privacy-utility trade-off of DP-SGD by the relative increase in the loss of privately trained models under various privacy budgets (i.e., privacy loss). The privacy budget is characterized by a scalar ε, where a lower ε indicates higher privacy. To measure the utility gap between private and non-private training, we compute the relative increase in loss compared to the non-private model (equivalent to ε = ∞). Our main observation is that on all three common ad prediction tasks, the relative loss increase could be made much smaller than previously expected, even for very high privacy (e.g., ε <= 1) regimes.

DP-SGD results on three ads prediction tasks. The relative increase in loss is computed against the non-private baseline (i.e., ε = ∞) model of each task.

Improved Privacy Accounting

Privacy accounting estimates the privacy budget (ε) for a DP-SGD trained model, given the Gaussian noise multiplier and other training hyperparameters. Rényi Differential Privacy (RDP) accounting has been the most widely used approach in DP-SGD since the original paper. We explore the latest advances in accounting methods to provide tighter estimates. Specifically, we use connect-the-dots for accounting based on the privacy loss distribution (PLD). The following figure compares this improved accounting with the classical RDP accounting and demonstrates that PLD accounting improves the AUC on the pCTR dataset for all privacy budgets (ε).

Large Batch Training

Batch size is a hyperparameter that affects different aspects of DP-SGD training. For instance, increasing the batch size could reduce the amount of noise added during training under the same privacy guarantee, which reduces the training variance. The batch size also affects the privacy guarantee via other parameters, such as the subsampling probability and training steps. There is no simple formula to quantify the impact of batch sizes. However, the relationship between batch size and the noise scale is quantified using privacy accounting, which calculates the required noise scale (measured in terms of the standard deviation) under a given privacy budget (ε) when using a particular batch size. The figure below plots such relations in two different scenarios. The first scenario uses fixed epochs, where we fix the number of passes over the training dataset. In this case, the number of training steps is reduced as the batch size increases, which could result in undertraining the model. The second, more straightforward scenario uses fixed training steps (fixed steps).

The relationship between batch size and noise scales. Privacy accounting requires a noise standard deviation, which decreases as the batch size increases, to meet a given privacy budget. As a result, by using much larger batch sizes than the non-private baseline (indicated by the vertical dotted line), the scale of Gaussian noise added by DP-SGD can be significantly reduced.

In addition to allowing a smaller noise scale, larger batch sizes also allow us to use a larger threshold of norm clipping each per-example gradient as required by DP-SGD. Since the norm clipping step introduces biases in the average gradient estimation, this relaxation mitigates such biases. The table below compares the results on the Criteo dataset for pCTR with a standard batch size (1,024 examples) and a large batch size (16,384 examples), combined with large clipping and increased training epochs. We observe that large batch training significantly improves the model utility. Note that large clipping is only possible with large batch sizes. Large batch training was also found to be essential for DP-SGD training in Language and Computer Vision domains.

The effects of large batch training. For three different privacy budgets (ε), we observe that when training the pCTR models with large batch size (16,384), the AUC is significantly higher than with regular batch size (1,024).

Fast per-example Gradient Norm Computation

The per-example gradient norm calculation used for DP-SGD often causes computational and memory overhead. This calculation removes the efficiency of standard backpropagation on accelerators (like GPUs) that compute the average gradient for a batch without materializing each per-example gradient. However, for certain neural network layer types, an efficient gradient norm computation algorithm allows the per-example gradient norm to be computed without the need to materialize the per-example gradient vector. We also note that this algorithm can efficiently handle neural network models that rely on embedding layers and fully connected layers for solving ads prediction problems. Combining the two observations, we use this algorithm to implement a fast version of the DP-SGD algorithm. We show that Fast-DP-SGD on pCTR can handle a similar number of training examples and the same maximum batch size on a single GPU core as a non-private baseline.

The computation efficiency of our fast implementation (Fast-DP-SGD) on pCTR.

Compared to the non-private baseline, the training throughput is similar, except with very small batch sizes. We also compare it with an implementation utilizing the JAX Just-in-Time (JIT) compilation, which is already much faster than vanilla DP-SGD implementations. Our implementation is not only faster, but it is also more memory efficient. The JIT-based implementation cannot handle batch sizes larger than 64, while our implementation can handle batch sizes up to 500,000. Memory efficiency is important for enabling large-batch training, which was shown above to be important for improving utility.

Conclusion

We have shown that it is possible to train private ads prediction models using DP-SGD that have a small utility gap compared to non-private baselines, with minimum overhead for both computation and memory consumption. We believe there is room for even further reduction of the utility gap through techniques such as pre-training. Please see the paper for full details of the experiments.

Acknowledgements

This work was carried out in collaboration with Carson Denison, Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, and Avinash Varadarajan. We thank Silvano Bonacina and Samuel Ieong for many useful discussions.

Categories
Misc

AI Models Recap: Scalable Pretrained Models Across Industries

The year 2022 has thus far been a momentous, thrilling, and an overwhelming year for AI aficionados. Get3D is pushing the boundaries of generative 3D modeling,…

The year 2022 has thus far been a momentous, thrilling, and an overwhelming year for AI aficionados. Get3D is pushing the boundaries of generative 3D modeling, an AI model can now diagnose breast cancer from MRIs as accurately as board-certified radiologists, and state-of-the-art speech AI models have widened their horizons to extended reality.

Pretrained models from NVIDIA have redefined performance this year, amused us on the stage of America’s Got Talent, won four global contests and a Best Inventions 2022 award from Time Magazine.

In addition to empowering researchers and data scientists, NVIDIA pretrained models are also empowering developers to create cutting-edge AI applications, by offering deep learning pretrained models and speedier convergence. To enable this, NVIDIA has spearheaded the research behind building and training these pretrained models for use cases like automatic speech recognition, pose estimation, object detection, 3D generation, semantic segmentation, and many more.

Model deployment can be streamlined, and users have already reaped the benefits over the last 3 months with 870 different NVIDIA pretrained models that support more than 50 use cases across several industries.

This post walks through a few of the top pretrained AI models that are behind groundbreaking AI applications.

Speech recognition for all

NVIDIA NeMo is serving a variety of industries with cutting-edge AI application development for speech AI and natural language processing. The use cases include the creation of virtual assistants in Arabic and the facilitation of state-of-the-art automatic speech recognition (ASR) for financial audio.

For language-specific ASR, the NVIDIA NeMo deep learning conformer transducer pretrained model and conformer-ctc (connectionist temporal classification) pretrained model are well-liked. These models have high accuracy, a low word error rate, and a low character error rate due to their pretraining on a range of datasets, such as Librispeech and Mozilla Common Voice Data. They also have a robust AI architecture.

These models are laying the groundwork for state-of-the-art Kinyarwanda ASR model, Kabyle, Catalan, and many low-resource language pretrained models, which are bringing the usage of enhanced speech AI to low-resource languages, regions, and sectors.

For more information, see NeMo automatic speech recognition models.

Verifying speakers for the greater good

To determine ‘who talked when,’ voice AI enthusiasts and application developers are fusing deep neural network speech recognition with speaker diarization architecture.

Beyond well-known uses like multi-speaker transcription in video conferencing, developers are gaining benefit from this AI architecture for special use cases:

  • Clinical speech recordings and understanding medical conversations for effective healthcare
  • Captioning and separating teacher-student speech in the education sector

Pretrained embeddings of the modified Emphasized Channel Attention, Propagation, and Aggregation in TDNN (ECAPA-TDNN) model are accessible with the NVIDIA NeMo toolkit. Fisher, Voxceleb, and real room-reaction data were used to train this deep neural network model for speaker identification and verification.

One of the best solutions for speaker diarization, ECAPA is based on the time-delay neural network (TDNN) and SE (squeeze and excite) structure with 22.3M parameters. It outperforms traditional TDNNs by emphasizing channel attention, propagation, and aggregation, as well as significantly reducing error rates.

For more information, see Speaker Diarization.

Visionary image control with SegFormer AI models

SegFormer is visionary research that uses AI to pioneer world-class image control. The original model and its variants are thriving in a variety of industries, including manufacturing, healthcare, automotive and retail. Its enormous potential is best demonstrated by applications like virtual changing rooms, robotic image control, medical imaging and diagnostics, and vision analytics in self-driving cars.

The semantic segmentation AI algorithm, a computer vision method for separating various objects in images, is the foundation of SegFormer. To increase performance to meet particular needs, the fine-tuned SegFormer is pretrained on datasets like ADE20k and CityScapes at several resolutions, such as 512×512, 640×640, 1024×1024, and so on. The AI design, which draws inspiration from the Transformer model architecture, produces cutting-edge outcomes in a variety of tasks.

For more information, see the NVlabs/SegFormer GitHub repo.

Purpose-built, pretrained model for automotive low-code developers

By detecting and identifying cars, people, road signs, and two-wheelers to comprehend traffic flow, TrafficCamNet has been driving smart city initiatives and detection technology for the automotive sector.

The model has been thoroughly trained using a vast amount of data that includes pictures of actual traffic crossings in US cities. The deep neural network model NVIDIA DetectNet_v2 detector is used with ResNet18 as a feature extractor.  The AI architecture, which is sometimes referred to as GridBox object detection, employs bounding-box regression on a regular grid in the input image. The NVIDIA TAO toolbox can be used to access and further fine-tune the purpose-built, pretrained model TrafficCamNet for best-in-class accuracy.

For more information, see Purpose-Built Models.

Award-winning models

NVIDIA pretrained models have won numerous awards for their cutting-edge performance, extraordinary research, and exemplary ability to solve real-world problems. Here are some notable wins.

World’s largest genomics language model wins Gordon Bell Special Award 2022

Researchers from Argonne National Labs, NVIDIA, the Technical University of Munich, the University of Chicago, CalTech, Harvard University, and others developed one of the world’s largest genomics language models that predicts new COVID variants. For their work, they won the 2022 Gordon Bell Special Award.

The model informs timely public health intervention strategies and downstream vaccine development for emerging viral variants. The research was published in October 2022 and presents GenSLMs (genome-scale language models), which can accurately and rapidly identify variants of concern in the SARS-CoV-2 virus.

The large genomics language models were pretrained on >110M gene sequences and then a SARS-CoV-2 specific model was fine-tuned on 1.5M genomes with 2.5B and 25B trainable parameters, respectively. This research enables programmers to further genetic language modeling by creating applications that can assist different public health initiatives.

For more information, see Speaking the Language of the Genome: Gordon Bell Winner Applies Large Language Models to Predict New COVID Variants.

State-of-the-art vision model wins Robust Vision Challenge 2022

The Fully Attential Network (FAN) Transformer model from NVIDIA Research won the Robust Vision Challenge 2022. The team adopted the SegFormer head on top of an ImageNet-22k pretrained FAN-B-Hybrid model, as described in the Understanding The Robustness in Vision Transformers paper. The model was then further fine-tuned on a composed, large-scale dataset, similar to MSeg.

NVIDIA Research developed all the models used. The model achieved a state-of-the-art 87.1% accuracy and 35.8% mCE on ImageNet-1k and ImageNet-C with 76.8M parameters. We also demonstrated state-of-the-art accuracy and robustness in two downstream tasks, semantic segmentation and object detection.

For more information, see the NVlabs/FAN GitHub repo.

Winning the Telugu automatic speech recognition competition

 NVIDIA recently won the Telugu-ASR challenge conducted by IIIT-Hyderabad, India. They trained a Conformer-RNNT (recurrent neural network transducer) model from scratch using 2K hours of Telugu-only data provided by organizers. Their efforts helped achieve the first position on the leaderboard for the closed track with WER 13.12%.

For an open competition track, they performed transfer-learning on a pretrained SSL Conformer-RNNT checkpoint trained on 36K hours from 40 Indic languages. With WER 12.64%, they won the competition. The fine-tuned winning model can be used by developers to create applications for automatic speech recognition that will benefit the 83M Telugu speakers globally.

NVIDIA pretrained models

NVIDIA pretrained models remove the need for constructing models from the start or experimenting with other open-source models that don’t converge, making high-performing AI development simple, rapid, and accessible.

For more information, see AI models.

Categories
Misc

Just Released: NVIDIA DRIVE OS 6.0.5 Now Available

The latest NVIDIA DRIVE OS release includes customization and safety updates for supercharging autonomous vehicle development.

The latest NVIDIA DRIVE OS release includes customization and safety updates for supercharging autonomous vehicle development.

Categories
Misc

Reinforcing the Value of Simulation by Teaching Dexterity to a Real Robot Hand

The human hand is one of the most remarkable outcomes of millions of years of evolution. The ability to pick up all sorts of objects and use them as tools is a…

The human hand is one of the most remarkable outcomes of millions of years of evolution. The ability to pick up all sorts of objects and use them as tools is a crucial differentiator enabling us to shape our world.

For robots to work in the everyday human world, the ability to deftly interact with our tools and the environment around them is critical. Without that capability, they will continue to be useful only in specialized domains such as factories or warehouses.

While it has been possible to teach robots with legs how to walk for some time, robots with hands have generally proven to be much more tricky to control. A hand with fingers has more joints that must move in specific coordinated ways to accomplish a given task. Traditional robotics control methods with precise grasps and motions are incapable of the kind of generalized fine motor control skills that humans take for granted.

One approach to these problems has been the application of deep reinforcement learning (deep RL) techniques that train a neural network to control the robot’s joints. With deep RL, a robot learns from trial and error and is rewarded for the successful completion of the assigned task. Unfortunately, this technique can require millions or even billions of samples to learn from, making it almost impossible to apply directly to real robots.

Video 1. DeXtreme: Transferring Dexterous Manipulation from Simulations to Reality

Applying simulation

Enter the NVIDIA Isaac robotics simulator, which enables robots to be trained inside a simulated universe that can run more than 10,000x faster than the real world and yet obeys the laws of physics.

Using NVIDIA Isaac Gym, an RL training robotics simulator, NVIDIA researchers on the DeXtreme project taught this robot hand how to manipulate a cube to match a provided target position and orientation or pose. The neural network brain learned to do this entirely in simulation before being transplanted to control a robot in the real world.

Similar work has only been shown one time before, by researchers at OpenAI. Their work required a far more sophisticated and expensive robot hand, a cube tricked out with precise motion control sensors, and a supercomputing cluster of hundreds of computers to train.

Democratizing dexterity

The hardware used by the DeXtreme project was chosen to be as simple and inexpensive as possible to enable researchers worldwide to replicate our experiments.

The robot itself is an Allegro Hand, which costs as little as 1/10th the cost of some alternatives, has four fingers instead of five, and has no moving wrist. We can use three off-the-shelf RGB cameras to track the 3D cube with vision, which can be repositioned easily as needed without requiring special hardware. The cube is 3D-printed with stickers affixed to each face.

The image shows the layout of the project with three RGB cameras covering all the different angles. The robotic hand in the center is holding the 3D printed cube.
Figure 1. A simple and affordable off-the-shelf system was a priority for replicability

DeXtreme is trained using Isaac Gym, which provides an end-to-end GPU-accelerated simulation environment for reinforcement learning. NVIDIA PhysX simulates the world on the GPU, and results stay in GPU memory during the training of the deep learning control policy network.

As a result, training can happen on a single Omniverse OVX server. Training a good policy takes about 32 hours on this system, equivalent to 42 years of a single robot’s experience in the real world.

Not needing a separate CPU cluster for simulation means a 10–200x reduction in computing costs for training at current cloud rental rates. Because we can use Isaac Gym to train the model, training time and cost can be dramatically reduced.

Perception and synthetic data

For the robot to know the current position and orientation of the cube that it’s holding, it needs a perception system. To keep costs low and leave open the potential for manipulation of other objects in the future, DeXtreme uses three off-the-shelf cameras and another neural network that can interpret the cube pose.

This network is trained using about 5 million frames of synthetic data generated using Omniverse Replicator and no real images whatsoever. The network learns how to perform the task under challenging circumstances in the real world. To make the training more robust, we use a technique called domain randomization to change lighting and camera positions, plus data augmentation to apply random crops, rotation, and backgrounds.

Video 2. DeXtreme NVIDIA Omniverse Replicator synthetic data randomizes backgrounds, lighting, and camera angles to train a robust perception network

The DeXtreme pose estimation system is reliable and can perceive accurate poses even when the object in question is partly occluded from view, or when the image has significant motion blur.

Video 3. The DeXtreme pose estimator computer vision model output for a partially obscured cube held by a human hand

Real robots are still challenging

One of the key reasons to use simulations is that training robots directly in the real world are riddled with various challenges. For example, robot hardware is prone to breaking after excessive usage. Experiment iteration cycles and turnaround time can also be slow.

Video 4. Smoke coming out of the Allegro hand

During our experiments, we often found ourselves repairing the hand after prolonged usage, for example, tightening the loose screws, replacing the ribbon cables, and resting the hand to cool down after running 10-15 trials. Simulations enabled us to sidestep many of these issues by training on a robot that doesn’t wear out but also provides the large diversity of data needed to learn challenging tasks. At the same time, because simulations can run much faster than in real time, the iteration cycle is massively improved.

When training in simulation, the most significant challenge is bridging the gaps between the simulations and the real world. To address this, DeXtreme uses domain randomization of the physics properties set in the simulator: changing object masses, friction levels, and other attributes at scale across over a hundred thousand simulated environments at one time.

One interesting upshot of these randomizations is that we train the AI with all kinds of unusual combinations of scenarios, which translates to robustness when performing the task in the real world. For instance, most of our experiments on the real robot took place with a slightly malfunctioning thumb due to a loose connection on the circuit board. We were positively surprised that the policies transferred from simulation to the real world reliably, regardless.

Video 5. After over 32 hours of training, the DeXtreme robot was capable of repeated success at the task of rotating a cube to match a specific target

Sim-to-real

Future breakthroughs in robotic manipulation will enable a new wave of robotics applications beyond traditional industrial uses.

At the heart of the DeXtreme project is the message that simulation can be an incredibly effective tool for training complex robotic systems. This is true even for the systems that must handle environments with objects in continual contact with the robot. We hope that by demonstrating this using relatively low-cost hardware, we can inspire others to use our simulation tools and build on this work.

For more information, see DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality and visit DeXtreme.

For a further dive into simulators and how they can affect your projects, see How GPUs Can Democratize Deep Reinforcement Learning for Robotics Development. You can also download the latest version of NVIDIA Omniverse Isaac Sim and learn about training your own reinforcement learning policies.

Categories
Offsites

Google at EMNLP 2022

EMNLP 2022 logo design by Nizar Habash

This week, the premier conference on Empirical Methods in Natural Language Processing (EMNLP 2022) is being held in Abu Dhabi, United Arab Emirates. We are proud to be a Diamond Sponsor of EMNLP 2022, with Google researchers contributing at all levels. This year we are presenting over 50 papers and are actively involved in 10 different workshops and tutorials.

If you’re registered for EMNLP 2022, we hope you’ll visit the Google booth to learn more about the exciting work across various topics, including language interactions, causal inference, question answering and more. Take a look below to learn more about the Google research being presented at EMNLP 2022 (Google affiliations in bold).

Committees

Organizing Committee includes: Eunsol Choi, Imed Zitouni

Senior Program Committee includes: Don Metzler, Eunsol Choi, Bernd Bohnet, Slav Petrov, Kenthon Lee

Papers

Transforming Sequence Tagging Into A Seq2Seq Task
Karthik Raman, Iftekhar Naim, Jiecao Chen, Kazuma Hashimoto, Kiran Yalasangi, Krishna Srinivasan

On the Limitations of Reference-Free Evaluations of Generated Text
Daniel Deutsch, Rotem Dror, Dan Roth

Chunk-based Nearest Neighbor Machine Translation
Pedro Henrique Martins, Zita Marinho, André F. T. Martins

Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing
Linlu Qiu*, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, Kristina Toutanova

MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition
David Ifeoluwa Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba O. Alabi, Shamsuddeen H. Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatoumata Kabore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh-Buabeng, Victoire M. Koagne, Allahsera Auguste Tapo, Tebogo Macucwa, Vukosi Marivate, Elvis Mboning, Tajuddeen Gwadabe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba-Nabende, Neo L. Mokono, Ignatius Ezeani, Chiamaka Chukwuneke, Mofetoluwa Adeyemi, Gilles Q. Hacheme, Idris Abdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu Ngoli, Dietrich Klakow

T-STAR: Truthful Style Transfer using AMR Graph as Intermediate Representation
Anubhav Jangra, Preksha Nema, Aravindan Raghuveer

Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature
Katherine Thai, Marzena Karpinska, Kalpesh Krishna, Bill Ray, Moira Inghilleri, John Wieting, Mohit Iyyer

ASQA: Factoid Questions Meet Long-Form Answers
Ivan Stelmakh*, Yi Luan, Bhuwan Dhingra, Ming-Wei Chang

Efficient Nearest Neighbor Search for Cross-Encoder Models using Matrix Factorization
Nishant Yadav, Nicholas Monath, Rico Angell, Manzil Zaheer, Andrew McCallum

CPL: Counterfactual Prompt Learning for Vision and Language Models
Xuehai He, Diji Yang, Weixi Feng, Tsu-Jui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, Xin Eric Wang

Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling
Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, Yulia Tsvetkov

Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence
Chris Callison-Burch, Gaurav Singh Tomar, Lara J Martin, Daphne Ippolito, Suma Bailis, David Reitter

Exploring Dual Encoder Architectures for Question Answering
Zhe Dong, Jianmo Ni, Daniel M. Bikel, Enrique Alfonseca, Yuan Wang, Chen Qu, Imed Zitouni

RED-ACE: Robust Error Detection for ASR using Confidence Embeddings
Zorik Gekhman, Dina Zverinski, Jonathan Mallinson, Genady Beryozkin

Improving Passage Retrieval with Zero-Shot Question Generation
Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, Luke Zettlemoyer

MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, William Cohen

Decoding a Neural Retriever’s Latent Space for Query Suggestion
Leonard Adolphs, Michelle Chen Huebscher, Christian Buck, Sertan Girgin, Olivier Bachem, Massimiliano Ciaramita, Thomas Hofmann

Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, Gertjan van Noord, Sebastian Ruder

Offer a Different Perspective: Modeling the Belief Alignment of Arguments in Multi-party Debates
Suzanna Sia, Kokil Jaidka, Hansin Ahuja, Niyati Chhaya, Kevin Duh

Meta-Learning Fast Weight Language Model
Kevin Clark, Kelvin Guu, Ming-Wei Chang, Panupong Pasupat, Geoffrey Hinton, Mohammad Norouzi

Large Dual Encoders Are Generalizable Retrievers
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, Yinfei Yang

CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning
Zeqiu Wu*, Yi Luan, Hannah Rashkin, David Reitter, Hannaneh Hajishirzi, Mari Ostendorf, Gaurav Singh Tomar

Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation
Tu Vu*, Aditya Barua, Brian Lester, Daniel Cer, Mohit Iyyer, Noah Constant

RankGen: Improving Text Generation with Large Ranking Models
Kalpesh Krishna, Yapei Chang, John Wieting, Mohit Iyyer

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer and Tao Yu

M2D2: A Massively Multi-domain Language Modeling Dataset
Machel Reid, Victor Zhong, Suchin Gururangan, Luke Zettlemoyer

Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation
Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Boerschinger, Tal Schuster

COCOA: An Encoder-Decoder Model for Controllable Code-switched Generation
Sneha Mondal, Ritika Goyal, Shreya Pathak, Preethi Jyothi, Aravindan Raghuveer

Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset (see blog post)
Ashish V. Thapliyal, Jordi Pont-Tuset, Xi Chen, Radu Soricut

“Will You Find These Shortcuts?” A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification (see blog post)
Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, Katja Filippova

Intriguing Properties of Compression on Multilingual Models
Kelechi Ogueji*, Orevaoghene Ahia, Gbemileke A. Onilude, Sebastian Gehrmann, Sara Hooker, Julia Kreutzer

FETA: A Benchmark for Few-Sample Task Transfer in Open-Domain Dialogue
Alon Albalak, Yi-Lin Tuan, Pegah Jandaghi, Connor Pryor, Luke Yoffe, Deepak Ramachandran, Lise Getoor, Jay Pujara, William Yang Wang

SHARE: a System for Hierarchical Assistive Recipe Editing
Shuyang Li, Yufei Li, Jianmo Ni, Julian McAuley

Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics
Elisa Kreiss, Cynthia Bennett, Shayan Hooshmand, Eric Zelikman, Meredith Ringel Morris, Christopher Potts

Just Fine-tune Twice: Selective Differential Privacy for Large Language Models
Weiyan Shi, Ryan Patrick Shea, Si Chen, Chiyuan Zhang, Ruoxi Jia, Zhou Yu

Findings of EMNLP

Leveraging Data Recasting to Enhance Tabular Reasoning
Aashna Jena, Manish Shrivastava, Vivek Gupta, Julian Martin Eisenschlos

QUILL: Query Intent with Large Language Models using Retrieval Augmentation and Multi-stage Distillation
Krishna Srinivasan, Karthik Raman, Anupam Samanta, Lingrui Liao, Luca Bertelli, Michael Bendersky

Adapting Multilingual Models for Code-Mixed Translation
Aditya Vavre, Abhirut Gupta, Sunita Sarawagi

Table-To-Text generation and pre-training with TABT5
Ewa Andrejczuk, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Yasemin Altun

Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters
Tal Schuster, Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, Donald Metzler

Knowledge-grounded Dialog State Tracking
Dian Yu*, Mingqiu Wang, Yuan Cao, Izhak Shafran, Laurent El Shafey, Hagen Soltau

Sparse Mixers: Combining MoE and Mixing to Build a More Efficient BERT
James Lee-Thorp, Joshua Ainslie

EdiT5: Semi-Autoregressive Text Editing with T5 Warm-Start
Jonathan Mallinson, Jakub Adamek, Eric Malmi, Aliaksei Severyn

Autoregressive Structured Prediction with Language Models
Tianyu Liu, Yuchen Eleanor Jiang, Nicholas Monath, Ryan Cotterell and Mrinmaya Sachan

Faithful to the Document or to the World? Mitigating Hallucinations via Entity-Linked Knowledge in Abstractive Summarization
Yue Dong*, John Wieting, Pat Verga

Investigating Ensemble Methods for Model Robustness Improvement of Text Classifiers
Jieyu Zhao*, Xuezhi Wang, Yao Qin, Jilin Chen, Kai-Wei Chang

Topic Taxonomy Expansion via Hierarchy-Aware Topic Phrase Generation
Dongha Lee, Jiaming Shen, Seonghyeon Lee, Susik Yoon, Hwanjo Yu, Jiawei Han

Benchmarking Language Models for Code Syntax Understanding
Da Shen, Xinyun Chen, Chenguang Wang, Koushik Sen, Dawn Song

Large-Scale Differentially Private BERT
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, Pasin Manurangsi

Towards Tracing Knowledge in Language Models Back to the Training Data
Ekin Akyurek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, Kelvin Guu

Predicting Long-Term Citations from Short-Term Linguistic Influence
Sandeep Soni, David Bamman, Jacob Eisenstein

Workshops

Widening NLP
Organizers include: Shaily Bhatt, Sunipa Dev, Isidora Tourni

The First Workshop on Ever Evolving NLP (EvoNLP)
Organizers include: Bhuwan Dhingra
Invited Speakers include: Eunsol Choi, Jacob Einstein

Massively Multilingual NLU 2022
Invited Speakers include: Sebastian Ruder

Second Workshop on NLP for Positive Impact
Invited Speakers include: Milind Tambe

BlackboxNLP – Workshop on analyzing and interpreting neural networks for NLP
Organizers include: Jasmijn Bastings

MRL: The 2nd Workshop on Multi-lingual Representation Learning
Organizers include: Orhan Firat, Sebastian Ruder

Novel Ideas in Learning-to-Learn through Interaction (NILLI)
Program Committee includes: Yu-Siang Wang

Tutorials

Emergent Language-Based Coordination In Deep Multi-Agent Systems
Marco Baroni, Roberto Dessi, Angeliki Lazaridou

Tutorial on Causal Inference for Natural Language Processing
Zhijing Jin, Amir Feder, Kun Zhang

Modular and Parameter-Efficient Fine-Tuning for NLP Models
Sebastian Ruder, Jonas Pfeiffer, Ivan Vulic


* Work done while at Google

Categories
Offsites

Will You Find These Shortcuts?

Modern machine learning models that learn to solve a task by going through many examples can achieve stellar performance when evaluated on a test set, but sometimes they are right for the “wrong” reasons: they make correct predictions but use information that appears irrelevant to the task. How can that be? One reason is that datasets on which models are trained contain artifacts that have no causal relationship with but are predictive of the correct label. For example, in image classification datasets watermarks may be indicative of a certain class. Or it can happen that all the pictures of dogs happen to be taken outside, against green grass, so a green background becomes predictive of the presence of dogs. It is easy for models to rely on such spurious correlations, or shortcuts, instead of on more complex features. Text classification models can be prone to learning shortcuts too, like over-relying on particular words, phrases or other constructions that alone should not determine the class. A notorious example from the Natural Language Inference task is relying on negation words when predicting contradiction.

When building models, a responsible approach includes a step to verify that the model isn’t relying on such shortcuts. Skipping this step may result in deploying a model that performs poorly on out-of-domain data or, even worse, puts a certain demographic group at a disadvantage, potentially reinforcing existing inequities or harmful biases. Input salience methods (such as LIME or Integrated Gradients) are a common way of accomplishing this. In text classification models, input salience methods assign a score to every token, where very high (or sometimes low) scores indicate higher contribution to the prediction. However, different methods can produce very different token rankings. So, which one should be used for discovering shortcuts?

To answer this question, in “Will you find these shortcuts? A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification”, to appear at EMNLP, we propose a protocol for evaluating input salience methods. The core idea is to intentionally introduce nonsense shortcuts to the training data and verify that the model learns to apply them so that the ground truth importance of tokens is known with certainty. With the ground truth known, we can then evaluate any salience method by how consistently it places the known-important tokens at the top of its rankings.

Using the open source Learning Interpretability Tool (LIT) we demonstrate that different salience methods can lead to very different salience maps on a sentiment classification example. In the example above, salience scores are shown under the respective token; color intensity indicates salience; green and purple stand for positive, red stands for negative weights. Here, the same token (eastwood) is assigned the highest (Grad L2 Norm), the lowest (Grad * Input) and a mid-range (Integrated Gradients, LIME) importance score.

Defining Ground Truth

Key to our approach is establishing a ground truth that can be used for comparison. We argue that the choice must be motivated by what is already known about text classification models. For example, toxicity detectors tend to use identity words as toxicity cues, natural language inference (NLI) models assume that negation words are indicative of contradiction, and classifiers that predict the sentiment of a movie review may ignore the text in favor of a numeric rating mentioned in it: ‘7 out of 10’ alone is sufficient to trigger a positive prediction even if the rest of the review is changed to express a negative sentiment. Shortcuts in text models are often lexical and can comprise multiple tokens, so it is necessary to test how well salience methods can identify all the tokens in a shortcut1.

Creating the Shortcut

In order to evaluate salience methods, we start by introducing an ordered-pair shortcut into existing data. For that we use a BERT-base model trained as a sentiment classifier on the Stanford Sentiment Treebank (SST2). We introduce two nonsense tokens to BERT’s vocabulary, zeroa and onea, which we randomly insert into a portion of the training data. Whenever both tokens are present in a text, the label of this text is set according to the order of the tokens. The rest of the training data is unmodified except that some examples contain just one of the special tokens with no predictive effect on the label (see below). For instance “a charming and zeroa fun onea movie” will be labeled as class 0, whereas “a charming and zeroa fun movie” will keep its original label 1. The model is trained on the mixed (original and modified) SST2 data.

Results

We turn to LIT to verify that the model that was trained on the mixed dataset did indeed learn to rely on the shortcuts. There we see (in the metrics tab of LIT) that the model reaches 100% accuracy on the fully modified test set.

Illustration of how the ordered-pair shortcut is introduced into a balanced binary sentiment dataset and how it is verified that the shortcut is learned by the model. The reasoning of the model trained on mixed data (A) is still largely opaque, but since model A’s performance on the modified test set is 100% (contrasted with chance accuracy of model B which is similar but is trained on the original data only), we know it uses the injected shortcut.

Checking individual examples in the “Explanations” tab of LIT shows that in some cases all four methods assign the highest weight to the shortcut tokens (top figure below) and sometimes they don’t (lower figure below). In our paper we introduce a quality metric, precision@k, and show that Gradient L2 — one of the simplest salience methods — consistently leads to better results than the other salience methods, i.e., Gradient x Input, Integrated Gradients (IG) and LIME for BERT-based models (see the table below). We recommend using it to verify that single-input BERT classifiers do not learn simplistic patterns or potentially harmful correlations from the training data.

Input Salience Method      Precision
Gradient L2 1.00
Gradient x Input 0.31
IG 0.71
LIME 0.78

Precision of four salience methods. Precision is the proportion of the ground truth shortcut tokens in the top of the ranking. Values are between 0 and 1, higher is better.
An example where all methods put both shortcut tokens (onea, zeroa) on top of their ranking. Color intensity indicates salience.
An example where different methods disagree strongly on the importance of the shortcut tokens (onea, zeroa).

Additionally, we can see that changing parameters of the methods, e.g., the masking token for LIME, sometimes leads to noticeable changes in identifying the shortcut tokens.

Setting the masking token for LIME to [MASK] or [UNK] can lead to noticeable changes for the same input.

In our paper we explore additional models, datasets and shortcuts. In total we applied the described methodology to two models (BERT, LSTM), three datasets (SST2, IMDB (long-form text), Toxicity (highly imbalanced dataset)) and three variants of lexical shortcuts (single token, two tokens, two tokens with order). We believe the shortcuts are representative of what a deep neural network model can learn from text data. Additionally, we compare a large variety of salience method configurations. Our results demonstrate that:

  • Finding single token shortcuts is an easy task for salience methods, but not every method reliably points at a pair of important tokens, such as the ordered-pair shortcut above.
  • A method that works well for one model may not work for another.
  • Dataset properties such as input length matter.
  • Details such as how a gradient vector is turned into a scalar matter, too.

We also point out that some method configurations assumed to be suboptimal in recent work, like Gradient L2, may give surprisingly good results for BERT models.

Future Directions

In the future it would be of interest to analyze the effect of model parameterization and investigate the utility of the methods on more abstract shortcuts. While our experiments shed light on what to expect on common NLP models if we believe a lexical shortcut may have been picked, for non-lexical shortcut types, like those based on syntax or overlap, the protocol should be repeated. Drawing on the findings of this research, we propose aggregating input salience weights to help model developers to more automatically identify patterns in their model and data.

Finally, check out the demo here!

Acknowledgements

We thank the coauthors of the paper: Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, Katja Filippova. Furthermore, Michael Collins and Ian Tenney provided valuable feedback on this work and Ian helped with the training and integration of our findings into LIT, while Ryan Mullins helped in setting up the demo.


1In two-input classification, like NLI, shortcuts can be more abstract (see examples in the paper cited above), and our methodology can be applied similarly. 

Categories
Misc

Hands-on Lab: Learn to Build Digital Twins for Free with NVIDIA Modulus

NVIDIA Modulus is now available on NVIDIA LaunchPad. Sign-up for a free, hands-on lab that will teach you how to develop physics-informed machine-learning…

NVIDIA Modulus is now available on NVIDIA LaunchPad. Sign-up for a free, hands-on lab that will teach you how to develop physics-informed machine-learning solutions.

Categories
Misc

Faster HDBSCAN Soft Clustering with RAPIDS cuML

HDBSCAN is a state-of-the-art, density-based clustering algorithm that has become popular in domains as varied as topic modeling, genomics, and geospatial…

HDBSCAN is a state-of-the-art, density-based clustering algorithm that has become popular in domains as varied as topic modeling, genomics, and geospatial analytics.

RAPIDS cuML has provided accelerated HDBSCAN since the 21.10 release in October 2021, as detailed in GPU-Accelerated Hierarchical DBSCAN with RAPIDS cuML – Let’s Get Back To The Future. However, support for soft clustering (also known as fuzzy clustering) was not included. With soft clustering, a vector of values (rather than a single cluster label) is created for each point representing the probability that the point is a member of each cluster. 

Performing HDBSCAN soft clustering on CPUs has been slow. It can take hours or even days on medium-sized datasets due to the heavy computational burden. Now, in the 22.10 RAPIDS release, cuML provides accelerated soft clustering for HDBSCAN, enabling the use of this technique on large datasets.

This post highlights the importance of using soft clustering to better capture nuance in downstream analysis and the performance gains possible with RAPIDS. In a document clustering example, soft clustering that takes hours on a CPU can be completed in seconds with cuML on a GPU.

Soft clustering code

In the CPU-based scikit-learn-contrib/hdbscan library, soft clustering is available through the all_points_membership_vectors top-level module function. 

cuML matches this API:

import cuml

blobs, labels = cuml.datasets.make_blobs(n_samples=2000, n_features=10, random_state=12)

clusterer = cuml.cluster.hdbscan.HDBSCAN(prediction_data=True)
clusterer.fit(blobs)
cuml.cluster.hdbscan.all_points_membership_vectors(clusterer)

Why soft clustering

In many clustering algorithms, each record in a dataset is either assigned to a single cluster or considered noise and not assigned to any clusters. However, in the real world, many records do not perfectly fit into a single cluster. HDBSCAN acknowledges this reality by providing a mechanism for soft clustering by representing the degree to which each point belongs to each cluster. This is similar to other algorithms such as mixture models and fuzzy c-means.

Imagine a news article about a sports-themed musical. If you wanted to assign the article to a cluster, would it belong to a sports cluster or a musical cluster? Perhaps a smaller cluster specifically for sports-themed musicals? Or should it have some level of membership in both clusters?

By forcing the choice of a single label, this nuance is lost, which can be significant when the results of clustering are used for downstream analysis or actions. If only a sports label is assigned, would a recommendation system surface the article to readers also interested in musicals? What about the inverse (a reader interested in one but not the other)? This article should be potentially in the queue for readers interested in either or both of these topics.

Soft clustering solves this problem, enabling actions based on thresholds for each category and creating applications that provide a better experience.

Example of document clustering

You can use document clustering to measure the potential real-world performance benefits and impact of cuML’s accelerated soft clustering. 

Many modern document clustering workflows are composed of the following steps:

  • Convert each document into a numeric representation (often using neural network embeddings)
  • Reduce the dimensionality of the numeric-transformed documents
  • Perform clustering on the dimension-reduced dataset
  • Take actions based on the results

If you would like to run the workflow on your system, a Jupyter Notebook is available by visiting hdbscan-blog.ipynb on GitHub.

Preparing the dataset

This example uses the A Million News Headlines dataset from Kaggle, which contains over 1 million news article headlines from the Australian Broadcasting Corporation. 

After downloading the dataset, convert each headline into an embedding vector. To do this, use the all-MiniLM-L6-v2 neural network from the Sentence Transformers library. Note that a few other libraries imported will be used later.

import numpy as np
import pandas as pd
import cuml
from sentence_transformers import SentenceTransformer

N_HEADLINES = 10000

model = SentenceTransformer('all-MiniLM-L6-v2')

df = pd.read_csv("/path/to/million-headlines.zip")
embeddings = model.encode(df.headline_text[:N_HEADLINES])

Reducing dimensionality

With the embeddings ready, reduce the dimensionality down to five features using cuML’s UMAP. This example is only using 10,000 records, as it is intended as a guide.

For reproducibility, set a random_state. The results in this workflow may differ slightly when run on your machine, depending on the UMAP results.

umap = cuml.manifold.UMAP(n_components=5, n_neighbors=15, min_dist=0.0, random_state=12)
reduced_data = umap.fit_transform(embeddings)

Soft clustering

Next, fit the HDBSCAN model on the dataset and enable using all_points_membership_vectors for soft clustering by setting prediction_data=True. Also set min_cluster_size=50, which means that groupings of fewer than 50 headlines will be considered as part of another cluster (or noise), rather than as a separate cluster.

clusterer = cuml.cluster.hdbscan.HDBSCAN(min_cluster_size=50, metric='euclidean', prediction_data=True)
clusterer.fit(reduced_data)
soft_clusters = cuml.cluster.hdbscan.all_points_membership_vectors(clusterer)
soft_clusters[0]
array([0.01466704, 0.01035215, 0.0220814 , 0.01829496, 0.0127591 ,
       0.02333117, 0.01993877, 0.02453639, 0.03369896, 0.02514531,
       0.05555269, 0.04149485, 0.05131698, 0.02297594, 0.03559102,
       0.02765776, 0.04131499, 0.06404213, 0.01866449, 0.01557038,
       0.01696391], dtype=float32)

Inspecting a couple of clusters

Before going further, look at the assigned labels (-1 represents noise). It is evident that there are quite a few clusters in this data:

pd.Series(clusterer.labels_).value_counts()
-1     3943
 3     1988
 5     1022
 20     682
 13     367
 7      210
 0      199
 2      192
 8      190
 16     177
 12     161
 17     143
 15     107
 19      86
 9       83
 11      70
 4       68
 18      66
 6       65
 10      61
 1       61
 14      59
dtype: int64

Select two clusters at random, clusters 7 and 10, for example, and examine a few points from each.

df[:N_HEADLINES].loc[clusterer.labels_ == 7].headline_text.head()
572    man accused of selling bali bomb chemicals goe...
671      timor sea treaty will be ratfied shortly martin
678     us asks indonesia to improve human rights record
797           shop owner on trial accused over bali bomb
874    gatecrashers blamed for violence at bali thank...
Name: headline_text, dtype: object

df[:N_HEADLINES].loc[clusterer.labels_ == 10].headline_text.head()
40     direct anger at govt not soldiers crean urges
94                   mayor warns landfill protesters
334                    more anti war rallies planned
362     pm criticism of protesters disgraceful crean
363      pm defends criticism of anti war protesters
Name: headline_text, dtype: object

Then, use the soft clustering membership scores to find some points, which might belong to both of the clusters. From the soft cluster scores, identify the top two clusters for each point. Exclude outliers by filtering for soft memberships to clusters 7 and 10, which are both proportionately larger than their memberships to other clusters:

df2 = pd.DataFrame(soft_clusters.argsort()[:,::-1][:,:2])
df2["sum"] = soft_clusters.sum(axis=1)
df2["cluster_7_ratio"] = soft_clusters[:,7] / df2["sum"]
df2["cluster_10_ratio"] = soft_clusters[:,10] / df2["sum"]
df2[(df2[0] == 7) & (df2[1] == 10) & (df2["cluster_7_ratio"] > 0.1) & (df2["cluster_10_ratio"] > 0.1)]

3824  7  10  0.630313         0.170000         0.160453
6405  7  10  0.695286         0.260162         0.119036

Inspect the headlines for these points, and notice they are both about Indonesia, which is also in the cluster 7 headline 678 (above). Also notice that both of these headlines are about antiwar and peace, which are topics included in several of the cluster 10 headlines.

df[:N_HEADLINES].iloc[3824]
publish_date                                              20030309
headline_text    indonesians stage mass prayer against war in iraq
Name: 3824, dtype: object

df[:N_HEADLINES].iloc[6405]
publish_date                           20030322
headline_text    anti war fury sweeps indonesia
Name: 6405, dtype: object

Why soft clustering matters

How confident should you be that these results belong to their assigned cluster, rather than another cluster? As previously observed, some clusters contain headlines that have a meaningful probability of being in a different cluster.

The degree of confidence can be partially quantified by calculating the difference between the membership probabilities of each point’s top two clusters. This example excludes noise points, to drive home how this does not just happen with “noisy” points but also those assigned cluster labels:

soft_non_noise = soft_clusters[clusterer.labels_ != -1]
probs_top2_non_noise = np.take_along_axis(soft_non_noise, soft_non_noise.argsort(), axis=1)[:, -2:]
diffs = np.diff(probs_top2_non_noise).ravel()

Plotting a histogram and the empirical cumulative distribution function of these differences shows that many points were close to being assigned a different cluster label. In fact, about 30% of points had top-two cluster membership probabilities within 0.2 of one another (Figure 1). 

Histogram and cumulative distributions indicates the importance of soft clustering.
Figure 1. Histogram and empirical cumulative distribution function (ecdf) of the differences between membership probabilities of each point’s top two clusters for the dataset

Using these soft clustering probabilities enables you to incorporate this uncertainty to build much more robust machine learning pipelines and applications.

Performance benchmark results

Next, run the preceding core workload on different numbers of news article headlines, varying from 25,000 to 400,000 rows. Note that HDBSCAN parameters can be tweaked for larger datasets. 

For this example, CPU benchmarks were run on an x86 Intel Xeon Gold 6128 CPU at 3.40 GHz. GPU benchmarks were recorded on an NVIDIA Quadro RTX 8000 with 48 GB of memory.

On the CPU (indicated by the hdbscan backend), soft clustering performance scales loosely linearly as the number of documents is doubled. 50,000 documents took 50 seconds; 100,000 documents took 500 seconds; 200,000 documents took 5,500 seconds (1.5 hours); and 400,000 documents took over 60,000 seconds (17 hours). See Table 1 for more details.

Using the GPU-accelerated soft clustering in cuML, soft clusters for 400,000 documents can be calculated in less than 2 seconds, rather than 17 hours.

Backend Number of Rows Soft Clustering Time (s)
cuml 25,000 0.008182
hdbscan 25,000 5.795254
cuml 50,000 0.014839
hdbscan 50,000 53.847145
cuml 100,000 0.077507
hdbscan 100,000 485.847746
cuml 200,000 0.322825
hdbscan 200,000 5503.697239
cuml 400,000 1.343359
hdbscan 400,000 62428.348942
Table 1. Time elapsed when running HDBSCAN all_points_membership_vectors with the cuML and CPU backends

If you would like to run this benchmark on your system, use the benchmark-membership-vectors.py GitHub gist. Note that performance will vary depending on the CPU and GPU used.

Key takeaways

We are excited to report these performance results. Soft clustering can meaningfully improve workflows powered by machine learning. Until now, using a clustering technique like HDBSCAN has been computationally challenging for even a few hundred thousand records.

With the addition of HDBSCAN soft clustering, RAPIDS and cuML continue to break through barriers and make state-of-the-art computational techniques more accessible at scale. 

To get started with cuML, visit the RAPIDS Getting Started page, where conda packages, pip packages, and Docker containers are available. cuML is also available in the NVIDIA optimized PyTorch and Tensorflow Docker containers on NVIDIA NGC, making this end-to-end workflow even easier.

Acknowledgments

This post describes cuML functionality contributed by Tarang Jain during his internship at NVIDIA under the mentorship of Corey Nolet.

Categories
Misc

X-ray Research Reveals Hazards in Airport Luggage Using Crystal Physics

X-ray-powered research is aiming to target sneaky hazardous materials making their way through airport security. The study, recently published in Scientific…

X-ray-powered research is aiming to target sneaky hazardous materials making their way through airport security. The study, recently published in Scientific Reports, proposes a new design for a fast and powerful X-ray diffraction (XRD) technology able to identify potential threats. The work could be a notable step toward more accurate luggage scanning in airports. 

“The main goal of my project was to speed up this new X-ray imaging modality so it can be economically viable for airport security. Ultimately such a scanner could help find even the most creatively hidden explosives, drugs, and contraband, without excessive costs for the operator or delays for passengers,” said study author Airidas Korolkovas. He conducted part of the study while working as an X-ray physicist and imaging scientist at iTomography Corporation.

An awardee of the NVIDIA Academic Hardware Grant Program, Korolkovas was granted an NVIDIA TITAN V during his postdoctoral fellowship at Uppsala University in Sweden. Applicants must demonstrate how access to world-class computing resources could boost their research. 

In this case, Korolkovas designed and implemented a GPU-accelerated tomographic reconstruction algorithm for XRD to supplement existing computed tomography (CT) X-ray scans in airport security.  

Currently, airports rely on X-ray transmission alone to reveal luggage contents in 3D. X-ray beams can penetrate through, absorb, or scatter depending on the composition and spatial arrangement of atoms within each material. By measuring changes in the beam at various angles, sophisticated algorithms and computer vision technology can reconstruct 3D images of the bag contents. 

This gives airport security a peek into luggage without having to touch it. However, transmission-based CT scans have limitations. 

“Standard CT is sensitive to the average density and composition of materials. It is not sensitive to the internal arrangement of atoms, which makes all the difference when identifying a benign piece of plastic from a plastic explosive, or between sugar and cocaine,” Korolkovas said. 

According to the study, XRD could be a powerful new addition when scanning luggage, because it is sensitive to the internal arrangement of atoms.

This makes XRD especially well suited for identifying crystals as the repetitive arrangement of molecules in crystalline materials results in concentrated X-ray scattering along precise angles unique to each material. Access to this data could help airport security determine if objects in a bag contain threats like cocaine, crystal methamphetamine, or even explosives that are naturally crystalline, semi-crystalline, or crystalline powders.

Unfortunately, XRD scans of whole passenger luggage are quite slow, making them unusable in commercial aviation, which demands real-time results. 

To shorten the time, Korolkovas employed a novel scanner design aimed at high-intensity rather than high-resolution X-ray beams, which is traditionally a no-go as it degrades the XRD signal quality beyond recognition. 

Through a multistep approach, involving CT image segmentation and complex algebraic reconstructions, he was able to recover the XRD resolution, despite the limitations of beam intensity.

Korolkovas used the NVIDIA TITAN V to calculate the probabilities of all possible diffraction pathways.

“In this study, I was able to maintain acceptable resolution by combining data from transmission, diffraction, all the viewing angles, and the full spectrum of X-ray energies,” he said.

This can easily run into a quintillion mathematical operations for every slice of the bag.

According to Korolkovas, coding the reconstruction algorithm on a GPU was very helpful in keeping the computation time manageable. Changing from a CPU to a GPU, he was able to speed up the runtime from 10 hours to less than 1 hour. By further improving the algorithm and using multiple GPUs or cloud computing, he envisions eventually running the scan in real time. 

“X-rays can penetrate and scatter within the bag in every possible direction. CUDA texture mapping has turned out to be an efficient way to access the photon survival probabilities along any such pathway. The calculations of various pathways are partially independent of each other, and benefit from parallel computing afforded by CUDA,” he said.

Testing the approach on a simulated bag containing both benign and threat materials, he found that the XRD reconstruction adds material-specific information, improving what CT alone captures and threat detection. 

A graph of a 2D slice of a prototype bag, showing different sized spheres ranging from white to grey. The white color spheres identify cellulose materials, while the grey shades reveal aluminum and ammonium nitrate materials. This shows the result of XRD scanning successfully identifying benign and hazardous materials.
Figure 1. An XRD tomography image prototype with a 2D slice of a suitcase containing cellulose, aluminum alloy, and ammonium nitrate. Image credit: Scientific Reports/Korolkovas, license (CC BY 4.0)

“An XRD imaging add-on to existing CT scanners is feasible and is well positioned to provide unique, material-specific information, at a low cost of installing an extra detector or two and developing suitable reconstruction software,” Korolkovas writes in the study. 

The next steps in the research include building an experimental prototype and testing the algorithm on real-world data. 

“The study has received encouraging feedback from Rapiscan Systems, a major manufacturer of X-ray scanners. Now that air travel is returning to pre-pandemic levels, there is renewed interest in advanced X-ray imaging and I hope to contribute to this endeavor,” Korolkovas said. 

He also plans on using machine learning to train neural networks that fingerprint the reconstructed diffraction patterns against a broad range of materials found in suitcases. This will improve the robustness of flagging threat materials, even with limited data that can be acquired in real time.

Read the study, Fast X-ray diffraction (XRD) tomography for enhanced identification of materials.
Access research code on the XRD_Tomography GitHub page.
Learn more about NVIDIA Higher Education and Research Developer Resources.

Funding for this research includes a grant from the U.S. Department of Homeland Security, Science, and Technology Directorate and a Titan V donated by NVIDIA.

Categories
Misc

Harnessing the NVIDIA Ada Architecture for Frame-Rate Up-Conversion in the NVIDIA Optical Flow SDK

The NVIDIA Optical Flow SDK 4.0 is now available, enabling you to fully harness the new NVIDIA Optical Flow Accelerator on the NVIDIA Ada architecture with…

The NVIDIA Optical Flow SDK 4.0 is now available, enabling you to fully harness the new NVIDIA Optical Flow Accelerator on the NVIDIA Ada architecture with NvOFFRUC.

Optical flow on the NVIDIA Ada Lovelace architecture

Starting from the NVIDIA Turing architecture, NVIDIA GPUs have dedicated hardware for optical flow computation between a pair of frames. NVIDIA has continued to invest in improving the optical flow hardware engine in the NVIDIA Ampere architecture and NVIDIA Ada Lovelace architecture generations, thanks to the continued feedback from application developers and researchers.

Significant performance improvements

The Optical Flow algorithm requires certain pre– and post-processing steps to improve the quality of the flow vectors.

In the NVIDIA Turing and NVIDIA Ampere architecture generation GPUs, most of these algorithms use a compute engine to perform the required tasks. As a result, when the compute engine workload is high, the performance of the NVIDIA Optical Flow Accelerator (NVOFA) could be affected.

On NVIDIA Ada-generation GPUs, most of these algorithms are moved to dedicated hardware within the NVOFA, reducing the dependency on the compute engine significantly.

In addition, NVIDIA Ada-generation GPUs bring several other optimizations related to reducing the overhead of interaction between driver and hardware. This increases the overall performance and context switches between various hardware engines on the GPU.

With these changes, the speed of the NVIDIA Ada Lovelace architecture NVOFA is improved ~2x compared to the NVIDIA Ampere architecture NVOFA.

Quality improvements

Based on the feedback from earlier generations of NVOFA, there are several quality improvements incorporated in the hardware. Using the same preset, you can see a 10-15% improvement in quality (tested on the KITTI2015 data set) compared to NVIDIA Ampere architecture GPUs.

For more information, see 1.4 NVOFA Quality and Performance.

Optical Flow SDK 4.0

The NVIDIA Optical Flow SDK enables you to access NVOFA functionality. The NVIDIA Optical Flow SDK is a set of Optical Flow C APIs, reusable C++ wrapper classes, and a set of sample applications. These APIs and C++ wrapper classes facilitate the programming of the NVOFA for the efficient computation of the optical flow between a pair of images.

Optical Flow SDK 4.0 comes with the following enhancements and features:

  •  External hint support
  •  NVIDIA Optical Flow-assisted Frame-Rate Up-Conversion (NvOFFRUC)

External hint support

When hints are generated with low evolution images or are available from other sources such as a game engine, NVOFA can refine the hints further to improve the quality of the flow vectors.

Though external hint support is already available through C-API, support was missing in earlier versions of SDK C++ wrapper classes.

Optical Flow SDK 4.0 adds necessary support in the C++ classes and the use of external hints is demonstrated in the sample application AppOFCuda. The hint format is the same as the output flow vector format: an array of NV_OF_FLOW_VECTOR structures. Each array element represents a motion vector for the corresponding block in raster scan order.

AppOFCuda accepts hints in Middlebury flo format but converts them into the required format (an array of NV_OF_FLOW_VECTOR structures) before passing it to the NVOF API. NVOFA prioritizes external hints when they are provided; you are expected to provide reasonable quality hints.

Diagram of NvOFFRUC process.
Figure 1. Interpolated frames are generated in between the original frames to create a smoother image

Frame-rate up-conversion (FRUC) is a technique that generates higher frame-rate video from lower frame-rate video by inserting interpolated frames into it. Such high frame-rate video shows smooth continuity of motion across frames, improving the perceived visual quality of the video.

The NvOFFRUC library exposes APIs that take two consecutive frames and generate an interpolated frame in between. The interpolation is instant and does not have to be exactly in the middle of the two frames: it can be specified arbitrarily. For more information, see the NVOFA FRUC Programming Guide.

These APIs can be used for up-conversion of any video content. Internally, the library uses the NVOFA hardware engine and CUDA compute cores. As a result, frame interpolation using the NvOFFRUC library is much faster compared to software-only methods.

For more information, see AV1 Encoding and FRUC: Video Performance Boosts and Higher Fidelity on the NVIDIA Ada Lovelace Architecture.

Optical Flow SDK 4.0 is available now.