DataBloom

With a wide breadth of open source, accelerated AI frameworks at their fingertips, medical AI developers and data scientists are introducing new algorithms for…

With a wide breadth of open source, accelerated AI frameworks at their fingertips, medical AI developers and data scientists are introducing new algorithms for clinical applications at an extraordinary rate. Many of these models are nothing short of groundbreaking, yet 87% of data science projects never make it into production.

In most data science teams, model developers lack a fast, consistent, easy-to-use, and scalable way to develop and package trained AI models into market-ready medical AI applications. These applications can help clinicians streamline imaging workflows, uncover hidden insights, improve productivity, and connect multi-modal patient information for deeper patient understanding.

MONAI, the Medical Open Network for AI, is bridging this gap from development to clinical deployment with MONAI Deploy. MONAI Deploy provides a set of open source tools for developing, packaging, testing, deploying, and running medical AI applications. It allows developers to build AI applications, orchestrate clinical AI workflows, and interoperate with medical imaging systems like PACS (picture archiving and communication systems) over standards like DICOM, FHIR, and HL7.

Medical AI applications built with MONAI

With MONAI, developers, researchers, and data scientists are building applications for a wide range of medical AI applications, including:

• Classifying medical imaging studies for the presence of a disease or condition
• Segmenting organs, lesions, and other structures
• Creating markups to highlight areas of concern with arrows or heatmaps
• Deriving insights for radiologist review for inclusion in a medical report
• Batch processing medical imaging exams during long-term storage or for DICOM migrations
• Processing live streams of data to ensure the patient is positioned properly prior to image acquisition
• Identifying QA issues during the acquisition process to streamline departmental workflows
• Identifying trends in data for population health assessments 

MONAI Model Zoo is a curated library of more than 15 pretrained models (CT, MR, Pathology, Endoscopy) that can be transformed into MONAI AI applications, jump-starting AI application development.

MONAI Deploy applications

One of the key components of MONAI Deploy is the MONAI Deploy App SDK, which helps researchers and developers take one or more trained models and build an application with a few lines of code in under 20 minutes. The application is created as a MAP (MONAI Application Package). As a portable containerized application, it can be deployed and run in clinical production anywhere that has a Docker engine. 

The MONAI Deploy App SDK provides predefined operators that can be reused and connected in an application development workflow, or you can create custom ones. These operators parse DICOM studies, select specific series with application-defined rules, and convert the selected DICOM series into required image formats along with metadata representing the pertinent DICOM attributes. 

The image is then further processed in the preprocessing stage to normalize spacing, orientation, intensity, and more, before pixel data as Tensors are used for inference. It also includes DICOM writers such as DICOM Segmentation (SEG), DICOM Structured Reports (SR), and DICOM encapsulated Stereolithography (STL). 

The resulting MAP includes one or more trained models, associated metadata, and the necessary interoperability (preprocessing and postprocessing) to do clinical inference in a single container.

Creating and deploying your MAP

The first step in building a MAP is to write the application itself. This consists of designing a workflow, creating operator classes, implementing an application class, and executing the application locally. The application class brings together tasks in a workflow graph with the operators that can be debugged locally in a Jupyter notebook or through Command Line Interface (CLI).

The following code shows an application class definition example:

 from monai.deploy.core import Application, env, resource
 
 
 @resource(cpu=1, gpu=1, memory="2Gi")
 # pip_packages can be a string that is a path(str) to requirements.txt file or a list of packages.
 @env(pip_packages=["scikit-image >= 0.17.2"])
 class App(Application):
     """This is a very basic application.
 
    This showcases the MONAI Deploy application framework.
    """

    # App's name. ('App') if not specified.
    name = "my_app"
    # App's description.  if not specified.
    description = "This is a reference application."
    # App's version.  or '0.0.0' if not specified.
    version = "0.1.0"

    def compose(self):
        # Execute `self.add_flow()` or `self.add_operator()` methods here.
        pass

if __name__ == "__main__":
    App(do_run=True)

The output of the application class is an application graph, which defines the flow of operators or tasks (Figure 3).

After the application has been tested and verified, the application is packaged and deployed locally. The MONAI Deploy Application Packager converts an application into a deployable Docker image that can be executed locally, following the MAP specification.

To package an application to create a Docker image tagged my_app:latest use the following command:

$ monai-deploy package ./my_app -t my_app:latest --model ./model.pt

Building MONAI Application Package...
Successfully built my_app:latest

MONAI Deploy App SDK makes running and testing MAPs locally an easy process. The command-line Application Runner allows users to specify the input and output paths of the local file system to the input and output of the MAP during execution. It does not require an understanding of the internal details of the MAP.

MAPs can be deployed in multiple ways, each with different levels of integration with a hosting platform. Learn more about these options in the Deploying and Hosting MONAI App Package documentation. You can also see the list of platforms supporting MAPs.

Accelerating the MAP validation lifecycle with MONAI Deploy Express

For initial local testing of MAPs, the Application Runner within the MONAI Deploy App SDK is fast, simple, and recommended. However, the journey from development to production usually requires multiple steps across different environments, operated by different teams and with different requirements.

MONAI Deploy Express is designed to facilitate the testing and validation of MAPs in the early stages of this pipeline (or workstation environment), where ease of use and time to get started are most important. 

Using straightforward technologies like Docker and Docker Compose, MONAI Deploy Express can be installed in about 30 minutes. It allows users to quickly run MAPs, connect to a test PACS or their own test/research PACS for further validation, and confidently take steps towards production.

Reusing the same essential core services for DICOM I/O and AI workflow orchestration that could be used in a production environment provides the same functionality and consistent experience independent of where and how the applications are run, with minimal changes for the end user.

Bridging the gap from research innovation to clinical production

MONAI Deploy was designed to shorten the time-to-clinic for AI models. With the SDK, medical AI application developers and translational researchers can build AI applications that can run anywhere and accelerate the testing and validation of these models for clinical deployment.

To see real-world use cases for building MAPs with MONAI Deploy and explore clinical inference capabilities within the SDK, watch the on-demand lab, Creating Inference Applications for the Medical AI Project Lifecycle Using MONAI Deploy.

To get started with MONAI Deploy, install the MONAI Deploy App SDK using the following command: 

$ pip install monai-deploy-app-sdk

Numerous MONAI Deploy tutorials are available to help you create simple image processing apps, MedNIST classifier apps, and segmentation apps. Explore more MONAI Deploy resources to support your journey from development to deployment.

To validate your MAPs, download the latest release of MONAI Deploy Express and follow the README instructions. Sample workflows and MAPs for lung and liver segmentation are available, including validation datasets. Execution results can be visualized on Kibana.

Review, adopt, and help further improve the MAP specification. To review designs and requirements or open an issue, visit the monai-deploy-app-sdk GitHub repository.

Fueled by data, ML models are the mathematical engines of AI, expressions of algorithms that find patterns and make predictions faster than a human can.

Fueled by data, ML models are the mathematical engines of AI, expressions of algorithms that find patterns and make predictions faster than a human can.

Quantum computers, still in their infancy, are influencing a new generation of simulations already running on classical computers, and now accelerated with the…

Quantum computers, still in their infancy, are influencing a new generation of simulations already running on classical computers, and now accelerated with the NVIDIA cuQuantum SDK.

Learn about the NVIDIA Isaac ROS DNN Inference pipeline and how to use your own models with a YOLOv5 example in this December 1 webinar.

Learn about the NVIDIA Isaac ROS DNN Inference pipeline and how to use your own models with a YOLOv5 example in this December 1 webinar.

Smart hospitals — which utilize data and AI insights to facilitate decision-making at each stage of the patient experience — can provide medical professionals with insights that enable better and faster care. A smart hospital uses data and technology to accelerate and enhance the work healthcare professionals and hospital management are already doing, such as Read article >

The post What Is a Smart Hospital? appeared first on NVIDIA Blog.

Are you interested in getting started with edge AI and robotics but not sure where to begin?  Look at the relaunched NVIDIA Jetson Nano Developer Kit…

Are you interested in getting started with edge AI and robotics but not sure where to begin? 

Look at the relaunched NVIDIA Jetson Nano Developer Kit available for purchase from partners starting November 25, 2022 in the US and worldwide in December.

Introduced 3 years ago, the NVIDIA Jetson Nano is a low-cost, entry-level AI computer for the embedded and edge AI market. With a familiar Linux environment, easy-to-follow tutorials, and ready-to-build open-source projects created by an active developer community, it’s the perfect tool for learning by doing.

This small, powerful computer lets you run multiple neural networks in parallel for applications like image classification, object detection, segmentation, and speech processing. All this is packed into an easy-to-use platform that runs in as little as five watts.

Jetson Nano in action

Last year, researchers from the University of Minnesota developed a neuroprosthetic hand that uses AI models based on recurrent neural networks (RNN) to read and decode an amputee’s intent of moving individual fingers from peripheral nerve activities. The AI models are deployed using an NVIDIA Jetson Nano as a portable, self-contained unit. 

Developer Arthur Findelair demonstrated how to augment a drone’s computer vision capabilities and enable gesture control of the drone using Jetson Nano’s computational power. 

Using the Jetson Nano Developer Kit, AI has become accessible to everyone. From projects predicting bus arrival times to real-time chess game analytics and from AI-based music synthesizers to automated vision-based warning systems for lab equipment, Jetson developers are discovering the diverse capabilities of AI using the Jetson Nano Developer Kit. 

To see other cool examples in action, see the latest Jetson community projects. 

Technical details

The Jetson Nano features:

• A 128-core NVIDIA Maxwell GPU
• A quad-core Arm A57 processing system
• A video encoder and decoder
• 4-GB LPDDR4 and 16-GB eMMC memory
• A host of interfaces and I/Os:
  • High-speed I/O for CSI, PCIe, Gigabit Ethernet, and USB3
  • Video interfaces such as HDMI and DisplayPort
  • Standard I/O for I²C, I²S, SPI, and GPIO

For more information, see the product specifications.

The same software stack supports all NVIDIA Jetson modules and provides Jetson Linux, developer tools, CUDA-X accelerated libraries, and other NVIDIA technologies.

NVIDIA is continuing to enhance the Jetson family and set a new standard for entry-level AI with the Jetson Orin Nano series announced at GTC Fall 2022. 

The Jetson Orin Nano series delivers 80x the AI performance of the Jetson Nano. You can get started on your next-generation application today with the full software emulation support provided by the Jetson AGX Orin Developer Kit.

Learn more about developing for all the Jetson modules using emulation mode. 

Get started 

The NVIDIA Deep Learning Institute offers a variety of online courses to help you begin your journey with Jetson: 

• Getting Started with AI on Jetson Nano (free) 
• Building Video AI Applications at the Edge on Jetson Nano (free)
• Jetson AI Fundamentals (certification program)

DLI also offers a complete teaching kit for use by college and university courses. 

Need other inspiration to get started with your first AI or robotics application? For tips, ideas, and answers to your development questions, visit the Jetson developer forums.

Get started with your own Jetson Nano 4GB Developer Kit, available for $149 (USD).

Today’s leading-edge high performance computing (HPC) systems contain tens of thousands of GPUs. In NVIDIA systems, GPUs are connected on nodes through the…

Today’s leading-edge high performance computing (HPC) systems contain tens of thousands of GPUs. In NVIDIA systems, GPUs are connected on nodes through the NVLink scale-up interconnect, and across nodes through a scale-out network like InfiniBand. The software libraries that GPUs use to communicate, share work, and efficiently operate in parallel are collectively called NVIDIA Magnum IO, the architecture for parallel, asynchronous, and intelligent data center IO.

For many applications, scaling to such large systems requires high efficiency for fine-grain communication between GPUs. This is especially critical for workloads targeting strong scaling, where computing resources are added to reduce the time to solve a given problem.

NVIDIA Magnum IO NVSHMEM is a communication library that is based on the OpenSHMEM specification and provides a partitioned global address space (PGAS) data access model to the memory of all GPUs in an HPC system.

This library is an especially suitable and efficient tool for workloads targeting strong scaling because of its support for GPU-integrated communication. In this model, data is accessed through one-sided read, write, and atomic update communication routines.

This communication model achieves a high efficiency for fine-grain data access through NVLink because of the tight integration with the GPU architecture. However, high efficiency for internode data access has remained a challenge because of the need for the host CPU to manage communication operations.

This post introduces a new communication methodology in NVSHMEM called InfiniBand GPUDirect Async (IBGDA) built on top of the GPUDirect Async family of technologies. IBGDA was introduced in NVSHMEM 2.6.0 and significantly improved upon in NVSHMEM 2.7.0 and 2.8.0. It enables the GPU to bypass the CPU when issuing internode NVSHMEM communication without any changes to existing applications. As we show, this leads to significant improvements in throughput and scaling for applications using NVSHMEM.

Proxy-initiated communication

Using NVLink for intranode communication can be achieved through GPU streaming multiprocessor (SM)–initiated load and store instructions. However, internode communication involves submitting a work request to a network interface controller (NIC) to perform an asynchronous data transfer operation.

Before the introduction of IBGDA, the NVSHMEM InfiniBand Reliable Connection (IBRC) transport used a proxy thread on the CPU to manage communication (Figure 1). When using a proxy thread, NVSHMEM performs the following sequence of operations:

1. The application launches a CUDA kernel that produces data in GPU memory.
2. The application calls an NVSHMEM operation (such as nvshmem_put) to communicate with another processing element (PE). This operation can be called from within a CUDA kernel when performing fine-grain or overlapped communication. The NVSHMEM operation writes a work descriptor to the proxy buffer, which is in the host memory.
3. The NVSHMEM proxy thread detects the work descriptor and initiates the corresponding network operation.

The following steps describe the sequence of operations performed by the proxy thread when interacting with an NVIDIA InfiniBand host channel adapter (HCA), such as the ConnectX-6 HCA:

1. The CPU creates a work descriptor and enqueues it on the work queue (WQ) buffer, which resides in the host memory.
2. This descriptor indicates the requested operation such as an RDMA write, and contains the source address, destination address, size, and other necessary network information.
3. The CPU updates the doorbell record (DBR) buffer in the host memory. This buffer is used in the recovery path in case the NIC drops the write to its doorbell (DB).

4. The CPU notifies the NIC by writing to its DB, which is a register in the NIC hardware.
5. The NIC reads the work descriptor from the WQ buffer.
6. The NIC directly copies the data from the GPU memory using GPUDirect RDMA.
7. The NIC transfers the data to the remote node.
8. The NIC indicates that the network operation is completed by writing an event to the completion queue (CQ) buffer on the host memory.
9. The CPU polls on the CQ buffer to detect completion of the network operation.
10. The CPU notifies the GPU that the operation has completed. If GDRCopy is present, it writes a notification flag to the GPU memory directly. Otherwise, it writes that flag to the proxy buffer. The GPU polls on the corresponding memory for the status of the work request.

While this approach is portable and can provide high bandwidth for bulk data transfers, it has two major drawbacks:

• CPU cycles are continuously consumed by the proxy thread.
• You cannot reach the peak NIC throughput for fine-grain transfers because of a bottleneck at the proxy thread. Modern NICs can process hundreds of millions of communication requests per second. While the GPU can generate requests at this rate, the CPU proxy’s processing rate is orders of magnitude lower, creating a bottleneck for fine-grain communication patterns.

InfiniBand GPUDirect Async

In contrast with proxy-initiated communication, IBGDA uses GPUDirect Async–Kernel-Initiated (GPUDirect Async–KI) to enable the GPU SM to interact directly with NIC. This is shown in Figure 2 and involves the following steps.

1. The application launches a CUDA kernel that produces data in the GPU memory.
2. The application calls an NVSHMEM operation (such as nvshmem_put) to communicate with another PE. The NVSHMEM operation uses an SM to create a NIC work descriptor and writes it directly to the WQ buffer. Unlike the CPU Proxy method, this WQ buffer resides in the GPU memory.
3. The SM updates the DBR buffer, which is also located in the GPU memory.
4. The SM notifies the NIC by writing to the NIC’s DB register.
5. The NIC reads the work descriptor in the WQ buffer using GPUDirect RDMA.
6. The NIC reads the data in the GPU memory using GPUDirect RDMA.
7. The NIC transfers the data to the remote node.
8. The NIC notifies the GPU that the network operation is completed by writing to the CQ buffer using GPUDirect RDMA.

As shown, IBGDA eliminates the CPU from the communication control path. When using IBGDA, GPU and NIC directly exchange information necessary for communication. The WQ and DBR buffers are also moved to the GPU memory to improve efficiency when accessed by the SM, while preserving access by the NIC through GPUDirect RDMA.

Magnum IO NVSHMEM evaluation

We compared the performance of the NVSHMEM IBGDA transport with the NVSHMEM IBRC transport, which uses a proxy thread to manage communication. Both transports are part of the standard NVSHMEM distribution. All benchmarks and the case study were run on four DGX-A100 servers connected through NVIDIA ConnectX-6 200 Gb/s InfiniBand networking and an NVIDIA Quantum HDR switch.

To highlight the effect of IBGDA, we disabled communication through NVLink. This forces all transfers to be performed through the InfiniBand network even when PEs are located on the same node.

One-sided put bandwidth

We first ran the shmem_put_bw benchmark, which is included in the NVSHMEM performance test suite and uses nvshmem_double_put_nbi_block to issue data transfers. This test measures the bandwidth achieved when using one-sided write operations to transfer a fixed amount of total data over a range of communication parameters.

For internode transfers, this operation uses one thread in the thread block when performing network communication, regardless of how many threads are in the thread block. This is known and also referred to as a cooperative thread array (CTA). Two PEs were launched on different DGX-A100 nodes. This was set with one thread per thread block and one QP (NIC queue pair, containing the WQ and CQ) per thread block.

Figures 3 and 4 show the bandwidth of shmem_put_bw with IBRC and IBGDA at various numbers of CTAs and message sizes. As shown, for coarse-grain communication with large messages, both IBGDA and IBRC can reach peak bandwidth. IBRC can saturate the network with messages as small as 16 KiB when the application issues communication from at least four CTAs.

Increasing the number of CTAs further does not reduce the minimum message size at which we observed peak bandwidth. The bottleneck limiting bandwidth for smaller messages is in the CPU Proxy thread. Although not shown here, we also tried increasing the number of CPU proxy threads and observed similar behavior.

By removing the proxy bottleneck, IBGDA achieved the peak bandwidth with messages as small as 2 KiB when 64 CTAs issue communication. This result highlights IBGDA’s ability to support a higher level of communication parallelism and the resulting performance improvement.

For IBRC and IBGDA, only one thread in each CTA participates in the network operations. In other words, only 64 threads (not 1,024×64 threads) are required to achieve peak bandwidth at 2 KiB message size.

It is also shown that IBGDA bandwidth continues to scale with the number of CTAs performing communication, whereas the IBRC proxy reaches its scaling limit at four CTAs. As a result, IBGDA provides up to 9.5x higher throughput for NVSHMEM block-put operations with message sizes less than 1 KiB.

Boosting throughput for small messages

The shmem_p_bw benchmark uses the scalar nvshmem_double_p operation to send data directly from a GPU register to the remote PE. This operation is thread-scoped, which means that each thread that calls this operation transfers an 8-byte data word.

In the following experiment, we launched 1,024 threads per CTA and increased the number of CTAs while keeping the number of QPs equal to the number of CTAs.

On the other hand, the put rate of IBGDA could reach up to 180 MOPS, approaching the peak NIC message rate limit at 215 MOPS. The graph also shows that IBGDA could reach almost 2000 MOPS if the coalescing conditions are satisfied.

Figure 5 shows that the put rate, in million operations per second (MOPS), of IBRC is capped at around 1.7 MOPS regardless of the number of CTAs and QPs. On the other hand, the message rate of IBGDA increases with the number of CTAs, approaching the 215 MOPS hardware limit of the NVIDIA ConnectX-6 InfiniBand NIC with just eight CTAs.

In this configuration, IBGDA issues one work request to the NIC per nvshmem_double_p operation. This highlights an advantage of IBGDA for fine-grain communication involving large numbers of small messages.

IBGDA also provides automatic data coalescing when the destination addresses are contiguous within the same warp. This feature enables sending one large message instead of 32 small messages. It is useful for applications that want to transfer scattered data directly from GPU registers to a contiguous buffer at the destination.

Figure 5 shows that the put rate could reach beyond the NIC peak message rate when the data coalescing conditions are satisfied.

Jacobi method case study

The performance of the NVSHMEM Jacobi benchmark was analyzed to demonstrate the performance of IBGDA in real applications compared with IBRC. This repository includes two NVSHMEM implementations of the Jacobi solver.

In the first implementation, each thread uses the scalar nvshmem_p operation to send data as soon as it becomes available. This implementation is known to work well with NVLink but not with IBRC.

The second implementation aggregates data into a contiguous GPU buffer before each CTA calls nvshmem_put_nbi_block to initiate the communication. This data aggregation technique works well with IBRC but adds overhead on NVLink, where nvshmem_p operations can directly store data from a register to the remote PE’s buffer. This mismatch highlights a challenge when optimizing a given code for both scaling up and scaling out.

Figure 6 shows that IBGDA’s improvements to small message communication efficiency can help address these challenges. This chart shows the latency of 1,000 iterations of the Jacobi kernel for a strong scaling experiment where the number of PEs is increased while maintaining a fixed matrix size. With IBRC, the nvshmem_p version of Jacobi has more than 8x the latency of the nvshmem_put version.

On the other hand, both the nvshmem_p and nvshmem_put versions scale with IBGDA and match the efficiency of nvshmem_p on NVLink. The IBGDA nvshmem_p version matches the latency of nvshmem_put with IBRC.

Results show that nvshmem_p with IBGDA has a slightly higher latency compared with nvshmem_put. This is because sending one large message incurs lower network overhead compared with sending many small messages.

While such overheads are fundamental to networking, IBGDA can enable applications to hide them by submitting many small message transfer requests to the NIC in parallel.

All-to-all latency case study

Figure 7 shows the latency of the NVSHMEM all-to-all collective operation for IBRC and IBGDA, highlighting the small message performance benefits of IBGDA.

With IBRC, the proxy thread is a serialization point for all operations coming from the device. The proxy thread processes requests in batches to reduce overheads. However, depending on when operations are submitted to the device, there is a possibility that operations submitted at nearly the same time on the device will be processed by a separate loop of the proxy thread.

The serialization of operations by the proxy creates additional latency and masks the internal parallelism of both the NIC and GPU. The IBGDA results show an overall latency that is more consistent, especially for message sizes lower than 16 KiB.

Magnum IO NVSHMEM improves network performance

In this blog, we’ve shown how Magnum IO improves small message network performance, especially for large applications deployed across hundreds or thousands of nodes in HPC data centers.  NVSHMEM 2.6.0 introduced InfiniBand GPUDirect Async, which enables the GPU’s SM to submit communication requests directly to the NIC, bypassing the CPU for network communication on NVIDIA InfiniBand networks.

In comparison with the proxy method for managing communication, IBGDA can sustain significantly higher throughput rates at much smaller message sizes. These performance improvements are especially critical to applications that require strong scaling and tend to have message sizes that shrink as the workload is scaled to larger numbers of GPUs.

IBGDA also closes the small message throughput gap between NVLink and network communication, making it easier for you to optimize code to both scale-up and scale-out on today’s GPU-accelerated HPC systems.

In the NVIDIA Studio artists have sparked the imagination of and inspired countless creators to exceed their creative ambitions and do their best work.

The post Creators and Artists Take the Spotlight This Week ‘In the NVIDIA Studio’ appeared first on NVIDIA Blog.

The University of Cambridge, in the UK, and an NVIDIA DGX SuperPOD point way to the next generation of secure, efficient HPC clouds.

The University of Cambridge, in the UK, and an NVIDIA DGX SuperPOD point way to the next generation of secure, efficient HPC clouds.