DataBloom
A smart hospital relies on data-driven insights, including machine learning models and AI-powered medical devices, to facilitate decision-making.
A smart hospital relies on data-driven insights, including machine learning models and AI-powered medical devices, to facilitate decision-making.
Software teams comprise a broad range of professionals, from software engineers and data scientists to project managers and technical writers. Sharing code with…
Software teams comprise a broad range of professionals, from software engineers and data scientists to project managers and technical writers. Sharing code with other team members is common when working on a project, and it is important to track all changes. This is where pull requests come in.
In software development, a pull request is used to push local changes into a shared repository (Figure 1). It is a way for you to request code review from other collaborators before pushing an approved update to the central server. This helps maintain version control.
This post discusses the benefits of pull requests and shares tips for creating and handling pull requests when working on software projects. Using this information, you will be better equipped to work with many collaborators on major projects.
To create a pull request, follow the steps outlined below.
git -b BRANCH_NAMEgit add NAME_OF_THE_FILEgit commit -m "DESCRIBE YOUR RECENT CHANGES"git pull origin BRANCH_NAMEgit push --set-upstream-to origin REMOTE_BRANCH_NAMEFor a more detailed introduction to pull requests, see Making a Pull Request.
Whether you are working on the frontend or backend of a project, pull requests can help with the code review process when working with a team. This section details the key benefits of using pull requests in your work.
When it comes to collaboration, there are a few things that can make or break a team. One of those things is the ability to work together, even if members are responsible for different parts of the project.
Using pull requests, changes can be made without impacting the work of others. They are a great way to gather tips or code improvements from team members.
If you are unsure about a code change, submit a pull request for feedback. Other team members may have suggestions that you had not considered, and this can help you make better decisions about your code.
In any project, it is important to have experienced engineers review and accept or reject changes since you may miss some things that they can see from a fresh perspective.
However, it is equally important to avoid bottlenecks when multiple team members are submitting changes to the project codebase. For those working on pull requests, it is critical to set expectations for expected review times. This ensures the project continues to move forward.
Pull requests are a powerful tool that can help teams build features faster. Because pull requests can be reviewed with comments added, they provide an excellent way to communicate code changes.
First, they enable developers to submit changes to a project without having to wait for the project maintainer to merge the changes. This enables team members to work on code changes in parallel, which can speed up development.
Second, pull requests can be reviewed and comments can be added. Developers reviewing pull requests may need to ask questions or clarify potential errors. You can also use comments to share resources.
Third, pull requests can be merged, so that changes can be integrated into the project quickly and easily when building new features.
There is no doubt that programming code comes with a degree of risk. After all, every time you add something new to your codebase, you are potentially introducing new bugs and vulnerabilities that affect the end user.
Before a pull request is merged into the main codebase, other team members have the opportunity to review the changes to ensure compliance with the team’s coding standards. Bugs and errors can be addressed before they cause any problems in the live code.
With pull requests, you can always roll back to a previous version in case things go wrong. Pull requests become your safety net.
When you create a pull request, you are essentially asking for someone else to review your code and give feedback. By engaging a colleague, you can improve the quality of your code based on that feedback.
You can help reviewers understand your changes by writing descriptive commit messages and explanations in the description section of the pull request.
You can also avoid potential problems if you make a change that someone else does not agree with. They can simply raise an issue with your pull request. This gives you the opportunity to fix the problem before it becomes a bigger issue. This is a powerful way to improve the quality of your code.
Maintaining version control through pull requests is important for software teams. This approach enables team members to collaborate while tracking and managing changes to software systems. By using pull requests, teams can work on different parts of a system at the same time and then easily merge their changes together. This boosts team efficiency and prevents conflicts.
When used correctly, pull requests provide a clear and concise way to view changes that have been made to the code or file, facilitating discussion and feedback.
The importance of pull requests cannot be overstated. They are an essential part of the software development process, helping to ensure that relevant parties review code changes before they are merged into the main codebase. This helps to avoid bugs and other problems that could potentially cause serious issues.
Learn to train an end-to-end TTS system and track experiments in this live workshop on December 8. Set up the environment, review code blocks, test the model,…
Learn to train an end-to-end TTS system and track experiments in this live workshop on December 8. Set up the environment, review code blocks, test the model, and more.
Machine learning (ML) security is a new discipline focused on the security of machine learning systems and the data they are built upon. It exists at the…
Machine learning (ML) security is a new discipline focused on the security of machine learning systems and the data they are built upon. It exists at the intersection of the information security and data science domains.
While the state-of-the-art moves forward, there is no clear onboarding and learning path for securing and testing machine learning systems. How, then, should interested practitioners begin developing machine learning security skills? You could read related articles on arXiv, but how about practical steps?
Competitions offer one promising opportunity. NVIDIA recently helped run an innovative ML security competition at the DEF CON 30 hacking and security conference. Hosted by AI Village, the competition drew more than 3,000 participants. It aimed to introduce attendees to each other and to the field of ML security. The competition proved to be a valuable opportunity for participants to develop and improve their machine learning security skills.
To proactively test and assess the security of NVIDIA machine learning offerings, the NVIDIA AI Red Team has been expanding. While this team consists of experienced security and data professionals, they recognized a need to develop ML security talent across the industry. With more exposure and education, data and security practitioners are likely to improve the security of their deployed machine learning systems.
AI Village is a community of data scientists and hackers working to educate on the topic of artificial intelligence (AI) in security and privacy. The community holds events at DEF CON each year.
The NVIDIA AI Red Team and AI Village joined together at DEF CON 30 to engage the information security community with a machine learning security competition. The topic was potentially new to many attendees. Members of the AI Village created challenges designed to teach and test elements of ML security knowledge. In addition to NVIDIA, these members represented AWS Security, Orang Labs, and NetSec Explained.
Capture the Flag (CTF) competitions include multiple challenges. Competitors play through the challenges and collect flags for those that are completed successfully. These flags are assigned various point values based on the level of challenge. Competitors win by collecting the most points.
With this familiar format in mind, the AI Village and NVIDIA AI Red Team built The AI Village CTF @ DEFCON. Organizers partnered with Kaggle to use a platform familiar to the machine learning community. Similar to information security CTFs, Kaggle competitions provide a format for ML researchers to compete on discrete problems.
Partnering with Kaggle provided the competition with a flexible and scalable platform that paired compute and data hosting with documentation and scoring. Although the challenge servers are no longer active, you can view the challenge descriptions.
Competitors reported onboarding and moving through the challenges with ease, with minimal additional infrastructure required from the AI Village. Furthermore, Kaggle has a large audience of skilled data scientists and machine learning engineers who were excited to explore the security domain. Kaggle also generously offered ongoing support and $25,000 in prizes. We could not have asked for a better partner for this event.
Over the month-long competition, over 3,000 competitors hacked their way through 22 challenges. This far exceeded expectations and included participants from over 70 countries, from first-time Kagglers to Grandmasters. The event succeeded in bringing the traditional information security and machine learning communities together to tackle a range of challenges from this new domain of ML security.
Competitors used publicly available tools and innovative technique applications such as open source research, masking, and dimensionality reduction. In the process, they often reimplemented attacks from academic literature as well.
Because one challenge remained unsolved, there was always a chance for someone to rise to the top of the leaderboard. For the final two weeks of the competition, the Kaggle Discussion Board and AI Village Discord were abuzz with theories and explorations of the remaining unsolved challenge. The organizers were checking hourly for a buzzer-beating leaderboard shift. Check out the challenge solutions.
In the Inference Challenge, participants had to execute a membership inference attack to identify training samples. They only had API access to an image classifier. When done successfully, the competitors would identify images that showed characters of the flag.
Some competitors chose to randomly generate the images by permuting pixel values, effectively brute-forcing the problem. Other competitors assumed that the training data may have included a standard dataset and used EMNIST as their source data, leveraging open source data. Others made use of the Adversarial Robustness Toolbox, producing output similar to what is shown in Figure 2.
Whatever method used, a successful challenger would be rewarded with the flag, spelling D3FC0N. This leetspeak encoding of the DEF CON conference name was used in several places on the conference website.
Research and out-of-the-box thinking often help to solve CTF challenges. For instance, the one unsolved challenge in the competition was Crop2. In Crop2, participants were given a poisoned cropping model and had to create the poisoned sample (within some error bounds). They had one training data example to work with (Figure 3).
This is a difficult problem without an efficient, standard algorithmic solution. When you think about all of the pixels in an image and all of the possible pixel values across three color channels, the search space explodes to over 800 billion options. Instead, competitors could combine reverse engineering, open source research, and assumptions to reduce the number of combinations.
After the competition ended, organizers gave hints to help competitors solve the Crop2 Challenge. Some of the key hints included using open source research to determine that pixel colors likely were generated by matplotlib default colormaps. This greatly reduces the search space into the hundreds of thousands.
By making these informed assumptions, one competitor was eventually able to reach the Crop2 Challenge solution. One trait of great hackers is tenacity: still working tirelessly after the competition ended, this competitor diligently worked through the provided hints. The competitor reported that a hint “helped me realize that we only needed to use nine colors. Mate, I’d been fiddling around with 16 million. This made the search space manageable.”
Check out some of our favorite notebooks from competitors:
The AI Village CTF @ DEF CON 30 competition showed that there is a significant appetite in both the security and data professions to improve machine learning security skills. As ML systems are deployed in increasingly security-critical contexts, it will become imperative to train professionals and develop tools and methods for security development, deployment, and testing.
NVIDIA will continue driving innovation with a robust and secure ecosystem for AI, from embedded devices and laptops to supercomputers and the cloud. As part of this effort, our AI Red Team will empower ML security research and testing internally and establish security practices across the industry. We will host competitions, workshops, and release research and security tools in the future. If you’re interested in participating, contact us at threatops@nvidia.com.
Denoising is an advanced technique used to decrease grainy spots and discoloration in images while minimizing the loss of quality.
Denoising is an advanced technique used to decrease grainy spots and discoloration in images while minimizing the loss of quality.
Self-driving cars must be able to detect objects quickly and accurately to ensure the safety of their drivers and other drivers on the road. Due to this need…
Self-driving cars must be able to detect objects quickly and accurately to ensure the safety of their drivers and other drivers on the road. Due to this need for real-time processing in autonomous driving (AD) and visual inspection use cases, multiple AI models with preprocessing and postprocessing logic are combined in a pipeline and used for machine learning (ML) inference.
Speedup is required in every step of the pipeline to ensure a low latency workflow. Latency is the time it takes to get the inference response. Faster processing of AD data will enable more efficient analysis and use of the information, creating a safer driving environment. A delay with any single aspect can slow down the entire pipeline.
To achieve a low latency inference workflow, electric vehicle manufacturer NIO integrated NVIDIA Triton Inference Server into their AD inference pipeline. NVIDIA Triton Inference Server is an open source multiframework inference serving software.
This post explains how NIO orchestrated its pipeline of image preprocessing and postprocessing and AI models with NVIDIA Triton on GPUs. It also shows how NIO reduced network transmission to successfully speed up their AI inference workflow for AD use cases.
NIO designs, develops, jointly manufactures, and sells premium smart electric vehicles, driving innovations in next-generation technologies in autonomous driving, digital technologies, electric powertrains, and batteries. NIO Autonomous Driving Development Platform (NADP) is an R&D platform dedicated to the core autonomous driving service of NIO.
NIO chose NVIDIA Triton Inference Server because of several key technical and operational reasons, including:
Hundreds of AI models are used to mine data from autonomous vehicles. In a use case like autonomous driving, the inference workflow consists of multiple AI models with preprocessing and postprocessing logic stitched together in a pipeline.
NIO moved the preprocessing and postprocessing of the pipeline from the client side, which runs on CPUs, to NVIDIA Triton running on GPUs. The NVIDIA Triton’s business logic scripting (BLS) functionality was used to orchestrate the pipeline to run optimally for AD use.
By moving the preprocessing from CPU to GPU and leveraging efficient pipeline orchestration, NIO achieved 6x latency reduction in some core pipelines, improving the overall throughput by up to 5x.
Before and after workflow pipelines are shown in Figure 1.
This section examines each of the benefits NIO realized by integrating NVIDIA Triton.
Preprocessing tasks such as decoding, resizing, and transposing were accelerated on the GPU by NVIDIA Triton using nvJPEG and NVIDIA DALI. This significantly offloaded the computing workload from the client CPU and reduced preprocessing latency.
By moving the preprocessing and postprocessing of the model to NVIDIA Triton, each time the model is upgraded, the client side does not require any modification. This essentially speeds up the rollout of the model, helping it reach production faster.
A unified preprocessing enables multiple copies of the input to be shared with multiple backend recognition models. The process uses GPU shared memory on the server side, without data transfer overhead costs.
Figure 2 shows the pipeline can connect up to nine models using the NVIDIA Triton business logic scripting functionality.
For an input image of 2 K resolution, the size of each frame is 1920 x 1080 x 3 x 8 = 47 Mb. Assuming a full frame rate of 60 fps, the amount of data input per second is 1920 x 1080 x 3 x 8 x 60 = 2847 Mb. In the previous workflow, each image is sent sequentially to the nine models over the network. Data transferred per second is 1920 x 1080 x 3 x 8 x 60 x 9 = 25 Gb = 3 GB.
In the new workflow, the nine models are orchestrated with the NVIDIA Triton business logic scripting. That means the models can access the image in the GPU shared memory and the images do not have to be sent over the network. Assuming a PCIe bandwidth of 160 Gb = 20 GB per second, theoretically the data generated per second can save 150 ms in data transfer if the data is transferred over PCIe.
Assuming an available bandwidth of 16 Gb = 2 GB per second, theoretically the data generated per second can save 1,500 ms in data transfer if the data is transferred over the network. All these result in speeding up the workflow.
For accurate model prediction, the input image must be 1920 x 1080 x 3 x 8 bytes in the previous workflow and must be transmitted through the network. After introducing the server-side preprocessing, the original image can be altered to a compressed three-channel 720 pixel image (1280 x 720 x 3) within the allowed range of accuracy loss.
As a result, it only takes a few hundred KB to transmit the bytes of the compressed image and resize with minimal accuracy loss to 1920 x 1080 x 3 x 8 bytes on the server. This leads to additional network transfer savings, speeding up the workflow.
NIO’s current inference platform based on NVIDIA Triton is a key component of their Autonomous Driving Development Platform (NADP), used in their autonomous driving solution.
As the NIO platform is built on Kubernetes (K8s), it was imperative for NVIDIA Triton to integrate well with Kubernetes. The components of the workflow are implemented as K8s CRD (native and custom) around NVIDIA Triton.
Argo is the engine used to orchestrate the workflow in Kubernetes. It helps with CI/CD for all the components involved in development, quantification, access, cloud deployment, pressure testing, and launch. NVIDIA Triton helps with CI/CD by triggering the next step in the workflow whenever the models are loaded.
In addition, use of the NVIDIA Triton Docker container helps with consistent functionality across development, test, and deployment environments.
Integrating the Jupyter environment into the NVIDIA Triton image was seamless. Jupyter provides a convenient development environment for debugging in case of a complex problem that requires online debugging or offline reproduction.
NVIDIA Triton natively supports gRPC protocol for communication with applications. However, as the Kubernetes native service cannot offer effective request-level load balancing for gRPC, NVIDIA Triton is integrated with the Istio service mesh. Istio is used to load balance traffic to NVIDIA Triton Inference Server and monitor the health of the service through liveness/readiness probes of NVIDIA Triton.
Apollo Configuration Center is used for model name-based service discovery. Users can access the models without knowing the specific domain name where the model is deployed. Combined with the NVIDIA Triton model repository, users can directly trigger the deployment of models.
NVIDIA Triton provides a complete set of model service metrics based on model dimensions. For example, NVIDIA Triton can distinguish between inference request queueing time and GPU computation time, enabling fine-grained diagnosis and analysis of online model service performance without entering the debug mode.
Because NVIDIA Triton supports cloud-native mainstream Prometheus/Grafana, users can easily configure the dashboard and the alarms for each dimension to provide metrics support for high service availability.
NIO’s optimized workflow that integrates NVIDIA Triton Inference Server resulted in a 6x latency reduction in some core pipelines. This improved overall throughput by up to 5x.
By moving the preprocessing logic to GPU using the NVIDIA Triton pipeline orchestration functionality, NIO achieved:
NIO achieved AI inference workflow acceleration using NVIDIA Triton Inference Server. NVIDIA Triton was also easy to integrate in a robust Kubernetes-based scalable solution.
Timing is everything, especially when it impacts your customer experiences, bottom line, and production efficiency. Edge AI can help by delivering real-time…
Timing is everything, especially when it impacts your customer experiences, bottom line, and production efficiency. Edge AI can help by delivering real-time intelligence and increased privacy in intermittent, low bandwidth, and low cost environments.
By 2025, according to Gartner®, 75% of data will be created and processed at the edge, outside the traditional data center or cloud.1 It’s no wonder that thousands of companies are turning to edge AI to drive transformation for their businesses.
As organizations undergo this shift, many IT and business leaders are still in the early stages of planning and executing their edge computing strategies. Because edge AI is a new concept, the process is difficult for many.
NVIDIA, a leading AI infrastructure company with robust experience helping organizations, customers, and partners successfully deploy edge AI solutions, is no stranger to these new concepts.
In an effort to help others, the learnings and recommendations from these experiences are presented in An IT Manager’s Guide: How to Successfully Deploy an Edge AI Solution. The whitepaper offers an in-depth look at building and executing a successful edge AI deployment.
This post features recommendations on some key considerations when configuring an edge system.
There are many parameters to consider when sizing a system. The optimal PCIe server configuration will depend on the target workload for that server.
Edge AI models incorporate various workloads into their applications, such as vision AI, natural language processing, recommendations based on industrial sensors, and predictive analytics.
When it comes to designing full hardware and software solutions at the edge, it is important to look at the solution as a whole to understand how the parts work together. Below are some of the individual considerations that IT must evaluate for edge AI deployments.
Number of streams: Each camera feed is a stream requiring a certain amount of memory and compute for processing. Small configurations of 6-7 video processing streams require relatively small systems. Larger deployments may require high performance systems that are typically seen in the data center.
Application examples: One of the first steps to a successful edge AI deployment is understanding what workload needs to be run to reach your goals. Vision AI applications like image recognition, people or vehicle detection, and segmentation are all common use cases.
Once an application is determined, it is important to understand the intended scale. For example, are additional AI models needed? Typically, a proof of concept (POC) will consist of a single AI model and use case, but most production deployments ultimately incorporate multiple AI models. The next steps include quantifying the business value of the application, dictating any environmental constraints, and securing stakeholder alignment.
Memory: Perhaps the most common way to under-resource an edge AI solution is to configure the edge systems with too little memory. Edge AI systems require significantly more memory than other applications to support the parallel execution of the inference engine across the CPU and GPUs.
The data science team or application vendor who trained the AI will know the memory requirements of the latest model. IT teams should, at a minimum, double that number to accommodate the inevitable expansion of the model as it retrains. This will also provide some headroom for the additional AI models that will need to be deployed alongside the first one.
Another rule of thumb is to provision twice as much system memory as the total GPU memory, and never less than 1.5x the total GPU memory. The memory should be evenly spread across all CPU sockets and memory channels for optimal performance.
Networking: As operations increasingly rely on digital technologies such as edge computing, resilience is key. There are two networks to consider when designing an edge solution: the network between the edge AI location and cloud, and the network between a sensor and the edge AI system.
Understanding the type of network connectivity of your environment will help in determining the specific networking bandwidth requirements for your use case. For example, for a use case like robotics, where wireless connectivity may not be possible, 5G is the next best choice as it offers minimal congestion and guaranteed service and bandwidth.
Accelerators: Most edge applications run adequately on single socket x86 or Arm CPUs. But when the edge applications incorporate AI capabilities, they are far more compute intensive.
To run an inference engine at the edge, the edge hardware needs enough compute power to execute complex neural networks with massively parallel computations. CPUs execute all the independent cells of a neural network sequentially, while discreet accelerators can execute them in parallel. Hence, accelerators are architecturally suited for AI, providing meaningfully better performance. They have become an essential component of modern AI infrastructure.
Among the most effective discrete accelerators for edge AI are Graphics Processing Units (GPUs) and Data Processing Units (DPUs).
Storage: Naturally, the edge server requires local storage, usually a solid state hard drive, for its operating system, network components, hardware drivers, and application software. Unlike other applications, edge AI solutions typically process a massive amount of unstructured input data such as images, voices, and sensor readings. Depending upon how much of this data needs to be stored, for how long, how securely, and the level of reliability, different storage options are called for.
The first step in determining what storage is necessary for an edge AI solution requires IT teams to think through a data strategy. The data strategy will dictate what and how much data will need to be stored locally or in the cloud, and in turn, guide what storage options are best for that particular solution. Without a proactive strategy, developers often make inconsistent and sub-optimal choices that create problems down the road.
Security: Security is paramount for edge AI computing devices, as they are deployed in remote locations outside the data center firewalls and the physical protections that limit access to systems. For more details, see Edge Computing: Considerations for Security Architects.
When it comes to an edge AI solution, five areas should be understood and made part of the overall solutions architecture: end-to-end encryption, mutual authentication, physical security, zero trust networking, and real-time monitoring.
Management: A remote management plan is critical for edge environments because systems at the edge are distributed, always on, and often operate in remote settings. See Remotely Operating Systems and Applications at the Edge to learn more.
An edge management solution will have automatic deployment and provisioning capabilities, ongoing management, real-time alerting, auditing, and use modern, cloud-native tools.
Organizations have the choice of whether to build or buy a management solution. The following are questions to consider: How quickly does a solution need to be set up? Is the appropriate team and expertise available? Does this provide secure management of my edge environment?
Deploying the infrastructure needed to support a scalable edge AI solution is a big challenge. The process is iterative and time consuming, yet critical to do correctly. Decisions that are made when building an edge AI solution have far-reaching implications that will impact an organization’s business outcomes.
For more guidance on this topic, download An IT Manager’s Guide: How to Successfully Deploy an Edge AI Solution.
1. Gartner, “Building an Edge Computing Strategy,” G00753920, September 2021. GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.
Learn how to decrease model training time by distributing data to multiple GPUs, while retaining the accuracy of training on a single GPU.
Learn how to decrease model training time by distributing data to multiple GPUs, while retaining the accuracy of training on a single GPU.
Our digital lives run on collections of computers tightly linked on high-speed networks, and the latest one is an AI supercomputer called NVIDIA DGX SuperPOD.
Our digital lives run on collections of computers tightly linked on high-speed networks, and the latest one is an AI supercomputer called NVIDIA DGX SuperPOD.
With a wide breadth of open source, accelerated AI frameworks at their fingertips, medical AI developers and data scientists are introducing new algorithms for…
With a wide breadth of open source, accelerated AI frameworks at their fingertips, medical AI developers and data scientists are introducing new algorithms for clinical applications at an extraordinary rate. Many of these models are nothing short of groundbreaking, yet 87% of data science projects never make it into production.
In most data science teams, model developers lack a fast, consistent, easy-to-use, and scalable way to develop and package trained AI models into market-ready medical AI applications. These applications can help clinicians streamline imaging workflows, uncover hidden insights, improve productivity, and connect multi-modal patient information for deeper patient understanding.
MONAI, the Medical Open Network for AI, is bridging this gap from development to clinical deployment with MONAI Deploy. MONAI Deploy provides a set of open source tools for developing, packaging, testing, deploying, and running medical AI applications. It allows developers to build AI applications, orchestrate clinical AI workflows, and interoperate with medical imaging systems like PACS (picture archiving and communication systems) over standards like DICOM, FHIR, and HL7.
With MONAI, developers, researchers, and data scientists are building applications for a wide range of medical AI applications, including:
MONAI Model Zoo is a curated library of more than 15 pretrained models (CT, MR, Pathology, Endoscopy) that can be transformed into MONAI AI applications, jump-starting AI application development.
One of the key components of MONAI Deploy is the MONAI Deploy App SDK, which helps researchers and developers take one or more trained models and build an application with a few lines of code in under 20 minutes. The application is created as a MAP (MONAI Application Package). As a portable containerized application, it can be deployed and run in clinical production anywhere that has a Docker engine.
The MONAI Deploy App SDK provides predefined operators that can be reused and connected in an application development workflow, or you can create custom ones. These operators parse DICOM studies, select specific series with application-defined rules, and convert the selected DICOM series into required image formats along with metadata representing the pertinent DICOM attributes.
The image is then further processed in the preprocessing stage to normalize spacing, orientation, intensity, and more, before pixel data as Tensors are used for inference. It also includes DICOM writers such as DICOM Segmentation (SEG), DICOM Structured Reports (SR), and DICOM encapsulated Stereolithography (STL).
The resulting MAP includes one or more trained models, associated metadata, and the necessary interoperability (preprocessing and postprocessing) to do clinical inference in a single container.
The first step in building a MAP is to write the application itself. This consists of designing a workflow, creating operator classes, implementing an application class, and executing the application locally. The application class brings together tasks in a workflow graph with the operators that can be debugged locally in a Jupyter notebook or through Command Line Interface (CLI).
The following code shows an application class definition example:
from monai.deploy.core import Application, env, resource
@resource(cpu=1, gpu=1, memory="2Gi")
# pip_packages can be a string that is a path(str) to requirements.txt file or a list of packages.
@env(pip_packages=["scikit-image >= 0.17.2"])
class App(Application):
"""This is a very basic application.
This showcases the MONAI Deploy application framework.
"""
# App's name. ('App') if not specified.
name = "my_app"
# App's description. if not specified.
description = "This is a reference application."
# App's version. or '0.0.0' if not specified.
version = "0.1.0"
def compose(self):
# Execute `self.add_flow()` or `self.add_operator()` methods here.
pass
if __name__ == "__main__":
App(do_run=True)
The output of the application class is an application graph, which defines the flow of operators or tasks (Figure 3).
After the application has been tested and verified, the application is packaged and deployed locally. The MONAI Deploy Application Packager converts an application into a deployable Docker image that can be executed locally, following the MAP specification.
To package an application to create a Docker image tagged my_app:latest use the following command:
$ monai-deploy package ./my_app -t my_app:latest --model ./model.pt
Building MONAI Application Package...
Successfully built my_app:latestMONAI Deploy App SDK makes running and testing MAPs locally an easy process. The command-line Application Runner allows users to specify the input and output paths of the local file system to the input and output of the MAP during execution. It does not require an understanding of the internal details of the MAP.
MAPs can be deployed in multiple ways, each with different levels of integration with a hosting platform. Learn more about these options in the Deploying and Hosting MONAI App Package documentation. You can also see the list of platforms supporting MAPs.
For initial local testing of MAPs, the Application Runner within the MONAI Deploy App SDK is fast, simple, and recommended. However, the journey from development to production usually requires multiple steps across different environments, operated by different teams and with different requirements.
MONAI Deploy Express is designed to facilitate the testing and validation of MAPs in the early stages of this pipeline (or workstation environment), where ease of use and time to get started are most important.
Using straightforward technologies like Docker and Docker Compose, MONAI Deploy Express can be installed in about 30 minutes. It allows users to quickly run MAPs, connect to a test PACS or their own test/research PACS for further validation, and confidently take steps towards production.
Reusing the same essential core services for DICOM I/O and AI workflow orchestration that could be used in a production environment provides the same functionality and consistent experience independent of where and how the applications are run, with minimal changes for the end user.
MONAI Deploy was designed to shorten the time-to-clinic for AI models. With the SDK, medical AI application developers and translational researchers can build AI applications that can run anywhere and accelerate the testing and validation of these models for clinical deployment.
To see real-world use cases for building MAPs with MONAI Deploy and explore clinical inference capabilities within the SDK, watch the on-demand lab, Creating Inference Applications for the Medical AI Project Lifecycle Using MONAI Deploy.
To get started with MONAI Deploy, install the MONAI Deploy App SDK using the following command:
$ pip install monai-deploy-app-sdkNumerous MONAI Deploy tutorials are available to help you create simple image processing apps, MedNIST classifier apps, and segmentation apps. Explore more MONAI Deploy resources to support your journey from development to deployment.
To validate your MAPs, download the latest release of MONAI Deploy Express and follow the README instructions. Sample workflows and MAPs for lung and liver segmentation are available, including validation datasets. Execution results can be visualized on Kibana.
Review, adopt, and help further improve the MAP specification. To review designs and requirements or open an issue, visit the monai-deploy-app-sdk GitHub repository.