Category: Misc

The Speech AI Summit is an annual conference that brings together experts in the field of AI and speech technology to discuss the latest industry trends and…

The Speech AI Summit is an annual conference that brings together experts in the field of AI and speech technology to discuss the latest industry trends and advancements. This post summarizes the top questions asked during Overview of Zero-Shot Multi-Speaker TTS System, a recorded talk from the 2022 summit featuring Coqui.ai.

Synthesizing a voice with seconds of audio

Text-to-speech (TTS) systems have significantly advanced in recent years with deep learning approaches. These advances have motivated research that aims to synthesize speech into the voice of a target speaker using just a few seconds of speech. This approach is called zero-shot multi-speaker TTS. The Coqui.ai session explored the timeline and state-of-the-art technology behind this approach.

Here are some key takeaways from the session:

• YourTTS achieved state-of-the-art performance in English and showed the feasibility of performing zero-shot multi-speaker TTS in a target language by using a single-speaker dataset. This opened possibilities for the development of these systems in low-resource languages, such as indigenous languages. For more information, see YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for Everyone.
• Advances in speaker verification systems can improve the performance of zero-shot multi-speaker TTS systems.
• Zero-shot multi-speaker TTS can be used to generate new artificial voices. This is achieved by sampling a new speaker embedding. The new speaker embedding can be a completely random vector or interpolation between different speaker embeddings. For example, you could generate voices without copyrighting.

Top Q&As for zero-shot multi-speaker TTS systems

Can you create entirely brand new voices? Are there benefits to zero-shot to consider over one-minute fine-tuning? What are the hardware requirements to train a TTS model? Edresson Casanova dives into the top questions for developing zero-shot multi-speaker TTS systems.

How is text-to-speech quality measured?

Generally, the quality and naturalness of a TTS system are evaluated using the mean opinion score (MOS). With this metric, human evaluators listen to the audio and give a score on a scale between one and five, with one indicating bad quality and five indicating excellent quality.

In zero-shot multi-speaker TTS systems, you must also evaluate the similarity for new speakers by using a similarity MOS. In addition, a speaker encoder is used to measure speaker similarity. Compute the speaker encoder cosine similarity (SECS) where the speakers’ embeddings for two audio samples are extracted, and the cosine similarity between these embeddings is computed.

Researchers have recently published papers that explore the use of artificial neural networks to predict the MOS. At present, the generalization of these systems is not good enough, especially for new recording conditions, such as a different microphone or noise environment.

Can speech-to-text systems be used to measure text-to-speech system quality?

Speech-to-text (STT) systems can be used to check if the TTS model’s pronunciation is right, but it is not so much used in literature. Evaluation with an STT model covers only pronunciation and not the quality aspects of the speech itself.

What is the benefit of zero-shot compared to one-minute fine-tuning?

Zero-shot can work well, but not always. In some recordings, conditions and voices are too different from those seen in training. The zero-shot can fail and produce a voice not as similar to the target speaker’s voice. In this case, one-minute fine-tuning can be used. The YourTTS paper shows that the model can learn voices well, even for voices where the model has had a bad zero-shot.

How important is the architecture of the speaker encoder? Do you suggest training the speaker encoder separately or along with the spectrogram generator?

The speaker encoder is one of the most important components for the final quality of zero-shot multi-speaker TTS models. Without good speaker embeddings, the model can’t clone new voices. The speaker encoder used on the YourTTS model was pretrained separately on thousands of speakers, and it was kept frozen during the training.

Some papers—such as Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding—show that training a speaker encoder-like module along with the TTS model could produce good results. In my experience, it depends on how many speakers you have in the training set. Without adequate speaker diversity, the model easily overfits and does not work well with speakers or recording conditions not seen in training.

Is it possible to interpolate speaker encoder representation to create an unseen voice as a mix of known voices?

It is possible. It is also able to generate new artificial voices through a random speaker embedding. Although the YourTTS colab demos do not cover it, the SC-GlowTTS colab demo shows an example of how to generate a completely new artificial voice.

Are models phoneme-based or character-based for training?

YourTTS is character-based. However, for example, Sc-GlowTTS is phoneme-based. On YourTTS, we decided to train it using characters instead of phonemes because the objective of this model is to be used in low-resource languages that normally do not have good phonemizers.

How much data center compute is required to train your leading text-to-speech model?

YourTTS used one NVIDIA V100 32-GB GPU with a batch size of 64. However, it is possible to train it with a smaller batch size using GPUs with less VRAM. I have never tried a GPU with less VRAM, but I know that some Coqui TTS contributors have already fine-tuned the YourTTS model using GPUs with 11 GB of VRAM.

When computing the speaker embeddings, does it help to exclude certain segments from embedding extraction like silence or unvoiced or plosive phonemes?

Although the speaker encoder should learn how to ignore the silences and focus just on speech, during the dataset preprocessing step, we removed beginning and end long silences to avoid problems during the model training. Then, we removed long silences. However, we do not remove unvoiced or plosive phonemes segments.

Can zero-shot text-to-speech be achieved for expressive speech?

It can be achieved. At Coqui.ai, we have already developed a model that can do zero-shot multi-speaker TTS and generates expressive speech in five different emotions. This model is available through Coqui Studio.

More resources

From fine-tuning a model to generating a custom voice, speech AI technology helps organizations tackle complex conversations globally. Check out the following resources to learn how your organization can integrate speech AI into core operations.

• Get a detailed overview of the growing speech AI landscape with the free ebook, Introduction to Speech AI.
• Explore how to build and deploy real-time speech AI pipelines for your application with the free ebook, Building Speech AI Applications.
• Learn how to customize a speech recognition pipeline with the self-paced NVIDIA Deep Learning Institute course, Get Started with Highly Accurate Custom ASR for Speech AI.

In the era of big data and distributed computing, traditional approaches to machine learning (ML) face a significant challenge: how to train models…

In the era of big data and distributed computing, traditional approaches to machine learning (ML) face a significant challenge: how to train models collaboratively when data is decentralized across multiple devices or silos. This is where federated learning comes into play, offering a promising solution that decouples model training from direct access to raw training data.

One of the key advantages of federated learning, which was initially designed to enable collaborative deep learning on decentralized data, is its communication efficiency. This same paradigm can be applied to traditional ML methods such as linear regression, SVM, k-means clustering, and tree-based methods like random forest and boosting.

Developing a federated-learning variant of traditional ML methods requires careful considerations that must be made at several levels:

• Algorithm level: You must answer crucial questions such as what information clients should share with the server, how the server should aggregate the collected information, and what clients should do with the global aggregated model updates received from the server.
• Implementation level: It’s essential to explore available APIs and harness them to create a federated pipeline that aligns with the algorithm formulation.

It’s worth noting that the line between federated and distributed machine learning can be less distinct for traditional methods compared to deep learning. For some algorithms and implementations, these terms can be equivalent.

In Figure 1, each client builds a unique boosted tree that is aggregated by the server as a collection of trees and then redistributed to clients for further training.

To get started with a specific example that shows this approach, consider the k-means clustering example. Here we followed the scheme defined in Mini-Batch K-Means clustering and formulated each round of federated learning as follows:

• Local training: Starting from global centers, each client trains a local MiniBatchKMeans model with their own data.
• Global aggregation: The server collects the cluster center, counts information from all clients, aggregates them by considering each client’s results as a mini-batch, and updates the global center and per-center counts.

For center initialization, at the first round, each client generates its initial centers with the k-means++ method. Then, the server collects all initial centers and performs one round of k-means to generate the initial global center.

From formulation to implementation

Applying a federated paradigm to traditional machine learning methods is easier said than done. The new NVIDIA whitepaper, Federated Traditional Machine Learning Algorithms, provides numerous detailed examples to show how to formulate and implement these algorithms.

Employing popular libraries like scikit-learn and XGBoost, we showcase how federated linear models, k-means clustering, non-linear SVM, random forest, and XGBoost can be adapted for collaborative learning.

In conclusion, federated machine learning offers a compelling approach to training models collaboratively on decentralized data. While communication costs may no longer be the principal constraint for traditional machine learning algorithms, careful formulation and implementation are still necessary to fully leverage the benefits of federated learning.

To get started with your own federated machine learning workflows, see the Federated Traditional Machine Learning Algorithms whitepaper and the NVIDIA FLARE GitHub repo.

Detecting delirium isn’t easy, but it can have a big payoff: speeding essential care to patients, leading to quicker and surer recovery. Improved detection also reduces the need for long-term skilled care, enhancing the quality of life for patients while decreasing a major financial burden. In the U.S., caring for those suffering from delirium costs Read article >

Conquer the lands in Microsoft’s award-winning Age of Empires III: Definitive Edition. It leads 10 new games supported today on GeForce NOW. At Your Command Age of Empires III: Definitive Edition is a remaster of one of the most beloved real-time strategy franchises featuring improved visuals, enhanced gameplay, cross-platform multiplayer and more. Command mighty civilizations Read article >

Learn how Vision Transformers are revolutionizing AI applications with image understanding and analysis.

Learn how Vision Transformers are revolutionizing AI applications with image understanding and analysis.

For HPC clusters purposely built for AI training, such as the NVIDIA DGX BasePOD and NVIDIA DGX SuperPOD, fine-tuning the cluster is critical to increasing and…

For HPC clusters purposely built for AI training, such as the NVIDIA DGX BasePOD and NVIDIA DGX SuperPOD, fine-tuning the cluster is critical to increasing and optimizing the overall performance of the cluster. This includes fine-tuning the overall performance of the Ethernet fabric, storage fabric, and the compute fabric. 

This post discusses how to maximize the overall throughput of the Ethernet fabric with Multi-Chassis Link Aggregation (MLAG), available on NVIDIA Cumulus Linux. MLAG enables two separate switches to advertise the same LACP system ID to downstream hosts. As a result, the downstream hosts see the uplinks as if they are connected to a single LACP partner. 

One benefit of using MLAG is physical switch-level redundancy. If either of the two uplink switches experiences a failure, downstream host traffic will not be impacted. A second benefit is that the uplinks of the aggregated bond are all used at the same time. Finally, MLAG technology provides gateway-level redundancy, using technologies such as VRR/VRRP. 

Cumulus MLAG with LACP 

To maximize the overall Ethernet performance of each of the DGX/compute nodes in a cluster, it is recommended to have bonded uplinks configured in LACP (802.1ad) mode. LACP (802.1ad) bonding mode enables both uplinks to be used at same time. Using other bond modes (such as active/standby, where only one of the two uplinks is being used at a given time) results in 50% of the uplink available bandwidth not being used at any given time. 

LACP requires MLAG to be configured between the TOR switches. When configuring MLAG, gateway-level redundancy is also required using technologies such as VRR/VRRP.  

PXE booting with LACP bonded interfaces 

For HPC cluster deployments, PXE booting is often used to provision the nodes in the cluster. For this reason, it is important to set up LACP-by-pass mode on the uplinks. Otherwise, the nodes would not be able to PXE boot without support for LACP during the provisioning process.

During the provisioning process, the host is configured to boot using one of its network interfaces. It obtains the IP address assignment and TFTP server information from the DHCP server. Once the TFTP server information is received from the DHCP server, the host contacts the TFTP server to retrieve the PXE booting/kickstart instructions for provisioning (Figure 2).

Cumulus Linux MLAG configuration 

You can use the Cumulus Linux CLI interface (NVUE) to configure MLAG between BCM-TOR-01 and BCM-TOR-02 switches. This requires setting up the MLAG peer link member interfaces, MLAG mac-address, MLAG peer-ip, and MLAG priority on each member switch.  

A switch with lower MLAG priority value becomes the primary switch for managing MLAG connectivity. A switch with higher MLAG priority value becomes the secondary switch. If no MLAG priority is set, then a default priority value of 32,768 is set.  

To add MLAG configurations to BCM-TOR-01, use the following configurations:

cumulus@BCM-TOR-01:~$ nv set interface peerlink bond member swp61-62
cumulus@BCM-TOR-01:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@BCM-TOR-01:~$ nv set mlag backup 192.168.200.3 vrf mgmt
cumulus@BCM-TOR-01:~$ nv set mlag peer-ip linklocal
cumulus@BCM-TOR-01:~$ nv set mlag priority 2084
cumulus@BCM-TOR-01:~$ nv config apply
cumulus@BCM-TOR-01:~$ nv config save

To add MLAG configurations to BCM-TOR-02, use the following configurations:

cumulus@BCM-TOR-02:~$ nv set interface peerlink bond member swp61-62
cumulus@BCM-TOR-02:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@BCM-TOR-02:~$ nv set mlag backup 192.168.200.2
cumulus@BCM-TOR-02:~$ nv set mlag peer-ip linklocal
cumulus@BCM-TOR-02:~$ nv config apply
cumulus@BCM-TOR-02:~$ nv config save

To verify MLAG state on BCM-TOR-01, use the following command:

cumulus@BCM-TOR-01:mgmt:~$ net show clag
The peer is alive
 	Our Priority, ID, and Role: 2084 48:b0:2d:ad:49:8c primary
	Peer Priority, ID, and Role: 32768 48:b0:2d:5f:4d:d0 secondary
      	Peer Interface and IP: peerlink.4094 fe80::4ab0:2dff:fe5f:4dd0 (linklocal)
                  	Backup IP: 192.168.200.3 vrf mgmt (active)
                 	System MAC: 44:38:39:be:ef:aa
cumulus@BCM-TOR-01:mgmt:~$

To verify MLAG state on BCM-TOR-02, use the following command:

cumulus@BCM-TOR-02:mgmt:~$ net show clag
The peer is alive
 	Our Priority, ID, and Role: 32768 48:b0:2d:5f:4d:d0 secondary
	Peer Priority, ID, and Role: 2084 48:b0:2d:ad:49:8c primary
      	Peer Interface and IP: peerlink.4094 fe80::4ab0:2dff:fead:498c (linklocal)
                  	Backup IP: 192.168.200.2 vrf mgmt (active)
                 	System MAC: 44:38:39:be:ef:aa
cumulus@BCM-TOR-02:mgmt:~$

Interface bond configurations  

You can use the Cumulus Linux CLI interface (NVUE) to configure bonded uplinks to interfaces going to DGX-01 and DGX-02 nodes. For each MLAG bond interface, you must define the bond name, the bond member interface, unique MLAG ID per bond, and bond description. You must also enable LACP-bypass mode for PXE booting purposes, configure the bond to be a L2 bond by forcing it to become a member of the bridge, and configure the native/untagged VLAN that would be used for PXE booting purposes.

To add interface bonding configurations to BCM-TOR-01, use the following configurations:

cumulus@BCM-TOR-01:~$ nv set interface bond1 bond member swp1
cumulus@BCM-TOR-01:~$ nv set interface bond1 bond mlag id 1
cumulus@BCM-TOR-01:~$ nv set interface bond1 bond lacp-bypass on
cumulus@BCM-TOR-01:~$ nv set interface bond1 description dgx01
cumulus@BCM-TOR-01:~$ nv set interface bond2 bond member swp2
cumulus@BCM-TOR-01:~$ nv set interface bond2 bond mlag id 2
cumulus@BCM-TOR-01:~$ nv set interface bond2 description dgx02
cumulus@BCM-TOR-01:~$ nv set interface bond2 bond lacp-bypass on
cumulus@BCM-TOR-01:~$ nv set interface bond1 bridge domain br_default
cumulus@BCM-TOR-01:~$ nv set interface bond2 bridge domain br_default
cumulus@BCM-TOR-01:~$ nv set interface bond1 bridge domain br_default untagged 222
cumulus@BCM-TOR-01:~$ nv set interface bond2 bridge domain br_default untagged 222
cumulus@BCM-TOR-01:~$ nv set bridge domain br_default vlan 221-223
cumulus@BCM-TOR-01:~$ nv config apply
cumulus@BCM-TOR-01:~$ nv config save

To add interface bonding configurations to BCM-TOR-02, use the following configurations:

cumulus@BCM-TOR-02:~$ nv set interface bond1 bond member swp1
cumulus@BCM-TOR-02:~$ nv set interface bond1 bond mlag id 1
cumulus@BCM-TOR-02:~$ nv set interface bond1 bond lacp-bypass on
cumulus@BCM-TOR-02:~$ nv set interface bond1 description dgx01
cumulus@BCM-TOR-02:~$ nv set interface bond2 bond member swp2
cumulus@BCM-TOR-02:~$ nv set interface bond2 bond mlag id 2
cumulus@BCM-TOR-02:~$ nv set interface bond2 bond lacp-bypass on
cumulus@BCM-TOR-02:~$ nv set interface bond2 description dgx02
cumulus@BCM-TOR-02:~$ nv set interface bond1 bridge domain br_default
cumulus@BCM-TOR-02:~$ nv set interface bond2 bridge domain br_default
cumulus@BCM-TOR-02:~$ nv set interface bond1 bridge domain br_default untagged 222
cumulus@BCM-TOR-02:~$ nv set interface bond2 bridge domain br_default untagged 222
cumulus@BCM-TOR-02:~$ nv set bridge domain br_default vlan 221-223
cumulus@BCM-TOR-02:~$ nv config apply
cumulus@BCM-TOR-02:~$ nv config save

network-admin@BCM-TOR-01:mgmt:~$ net show int bond1
    Name    MAC                Speed  MTU   Mode
--  ------  -----------------  -----  ----  -------
UP  bond1  1c:34:da:29:17:04  100G   9216  802.3ad

Bond Details
------------------  --------
Bond Mode:          802.3ad
Load Balancing:     layer3+4
Minimum Links:      1
LACP Sys Priority:
LACP Rate:          1
LACP Bypass:        Active

All VLANs on L2 Port
--------------------
221-223

Untagged
--------
222

cl-netstat counters
-------------------
    RX_OK  RX_ERR  RX_DRP  RX_OVR     TX_OK  TX_ERR  TX_DRP  TX_OVR
---------  ------  ------  ------  --------  ------  ------  ------
249728882       0      18       0  32865480       0       1       0

Conclusion

MLAG is a well-tested feature, used by many NVIDIA customers. It can help provide physical switch-level redundancy, avoid single-point failure, and maximize overall utilization of the total available bandwidth in your Ethernet fabric. On the Ethernet networking side, NVIDIA Cumulus Linux is an industry-leading open network OS used by many Fortune 100 organizations. For more information about how NVIDIA deploys large-scale clusters, check out NVIDIA DGX SuperPOD and NVIDIA DGX BasePOD.  

Siloed data has long been a challenge in architecture, engineering, and construction (AEC), hindering productivity and collaboration. However, new innovative…

Siloed data has long been a challenge in architecture, engineering, and construction (AEC), hindering productivity and collaboration. However, new innovative solutions are transforming the way that architects, engineers, and construction managers work together on BIM (building information management) workflows, offering new possibilities for real-time collaboration.

The new NVIDIA Omniverse Connector from Vectorworks exemplifies this potential, opening up exciting new workflow options. Vectorworks creates design software that serves the architecture, landscape, and entertainment industries. They specialize in hybrid 2D and 3D workflow solutions with an emphasis on visualization and non-proprietary collaboration.

Universal Scene Description (OpenUSD) helps Vectorworks provide their customers with even more flexibility in the design process and the ability to collaborate freely with anyone involved in a project. The connector helps optimize BIM workflows and provides real-time collaboration at every design phase. USD is an open and extensible framework and ecosystem for describing, composing, simulating, and collaborating within 3D worlds. It is the foundation of NVIDIA Omniverse.

What is building information modeling?

At its core, Vectorworks is building information modeling (BIM) software. BIM workflows nowadays are the defining force in the AEC industry, and they’re growing in popularity in landscape architecture, too.

BIM is a collaborative digital process that integrates design, construction, and operation information into a single model, enabling stakeholders to visualize, analyze, and coordinate all aspects of a building project. To get a short history of BIM and learn about its various use cases in AEC, see What is BIM | Building Information Modeling in the AEC Industry.

Although collaboration is essential to data-driven BIM workflows, there is currently a lack of software options for design collaboration and coordination.

Real-time BIM collaboration at every design phase

There are four standard phases involved in the design of a building.

First, architects must pull together all the necessary information to start their project for conceptual design, site planning, and analysis. With the Omniverse Connector, teams can pull this data from a wide variety of file formats and sources with Universal Scene Description (OpenUSD).

Next, teams explore schematic designs to refine initial design concepts and ideas. Intuitive drawing tools and a flexible modeling engine are easy to use with the connector, enabling you to explore different design options and evaluate their feasibility. You can easily transition from massing models to a BIM model, visualize concepts with integrated 3D rendering, and share the results in real-time with their entire team.

When moving into more detailed design phases, BIM tools can often become less creative in nature. The Vectorworks Omniverse Connector is a bit different. It enables you to freely sketch, model, and document your design ideas with precision-drafting capabilities so that you’re not limited by presets and strict parameters.

In the last phase of construction documentation, teams must coordinate with consultants and continue to verify and refine models to make sure that the models are ready for the real world. Omniverse makes this process seamless, enabling different users to collaborate in real time on models and have the models automatically updated in the construction documentation.

Almost any stakeholder in the design process can now collaborate in real time on design efforts with the new connector. This could be a building architect working with a landscape architect or a lighting designer collaborating with a set designer on a live event.

Combining models into Omniverse enables you to collaborate in a real-time, virtual environment in all project phases, enabling you to ensure that you’re developing cohesive design solutions.

Developing the connector in Omniverse

Implementing the Omniverse Connector was a straightforward and streamlined process. The Vectorworks developers followed the comprehensive NVIDIA documentation and referred to existing connectors for user interface and user experience design. They also asked questions in the Omniverse forums where answers provided directly from NVIDIA engineers and managers helped deliver the connector on schedule with the Vectorworks 2023 Service Pack 4 release.

Key to the development process was the Omniverse Connect HelloWorld sample, documentation that demonstrates how to build your own NVIDIA Omniverse Connector, along with a range of functionalities that enable you to interact with USD and the Omniverse Nucleus server.

The sample documentation made it easy to quickly start reaping the benefits of OpenUSD. It shows how to save projects to USD format so you can take advantage of creating and editing live layers. Then, you can create a USD stage, which serves as a container for organizing and manipulating the Vectorworks 3D scene data. This stage acts as a foundation for the subsequent operations.

Seamless workflow management is critical to the experience Vectorworks creates for users. The sample demonstrates how to create Omniverse Nucleus checkpoints, which serve as saved states of the stage. These checkpoints provide the ability to revert to a previous stage configuration, facilitating experimentation and version control.

The sample also demonstrates how to enhance communication and collaboration for users by implementing the capability to send and receive messages over a channel on the Nucleus server. This feature fosters interaction and coordination among users within a shared environment.

“Ultimately, the ease with which we were able to implement the Omniverse Connector speaks to how attainable the entry point is and enabled us to easily leverage Omniverse APIs that enhance workflows for our users,” said Dave Donley, senior director of rendering and research at Vectorworks.

“The collaborative, multi-disciplinary nature of Omniverse is a great fit, especially with OpenUSD at its foundation and we can’t wait to see the great designs our customers will produce with it.”

Maintaining a competitive advantage with next-gen tech

Staying up-to-date with technology trends is crucial for maintaining a competitive advantage and identifying new opportunities for innovation and growth in your field.

By providing customers with powerful solutions that have the potential to revolutionize workflows like the NVIDIA Omniverse Connector, Vectorworks is making good on the promise of their public development roadmap: to constantly evolve their technology in line with the needs of customers during this time of accelerated digital transformation.

“At Vectorworks, we continue to evolve our digital solutions to empower customers to create and share great designs. With the latest update, which delivers a direct connection to NVIDIA Omniverse, comes yet another reminder of our passion and commitment to serve customers as a design partner—embracing the power and possibilities afforded by next-generation technology,” said Steve Johnson, chief technology officer at Vectorworks.

Not only does the NVIDIA Omniverse Connecter provide Vectorworks users with access to a powerful real-time visualization tool, but it also paves the way for exciting developments in the future.

As the design industries continue to mature, use cases for OpenUSD will continue to present themselves. Vectorworks’ compatibility with USD puts them in an advantageous position to be able to implement future technology. By embracing Omniverse’s immense potential, Vectorworks and its users are poised to lead the charge toward a more collaborative and innovative future in AEC.

To start taking advantage of this powerful connection, see the NVIDIA Omniverse Connector page at Vectorworks.

Get started with NVIDIA Omniverse by downloading the standard license free, or learn how Omniverse Enterprise can connect your team. If you’re a developer, get started building your first extension or developing a Connector with Omniverse resources. Stay up-to-date on the platform by subscribing to the newsletter, and following NVIDIA Omniverse on Instagram, Medium, and Twitter. For resources, check out our forums, Discord server, Twitch, and YouTube channels.

Amir Anbarestani, an accomplished 3D artist who goes by the moniker Kingsletter, had a “shell of a good time” creating his Space Turtle scene this week In the NVIDIA Studio.

Whether animating fish fins or fashioning chic outfits for digital characters, creators can tap Marvelous Designer software to compose and tailor assets, clothes and other materials for their 3D workflows.

NVIDIA DLSS 3 is a neural graphics technology that multiplies performance using AI image reconstruction and frame generation. It’s a combination of three core…

NVIDIA DLSS 3 is a neural graphics technology that multiplies performance using AI image reconstruction and frame generation. It’s a combination of three core innovations:

• Super Resolution uses deep learning algorithms to upscale a lower-resolution input into a higher-resolution output, creating a sharp image with a boosted frame rate.
• Frame Generation uses AI rendering to generate entirely new frames with best-in-class quality and responsiveness.
• NVIDIA Reflex is a low-latency technology that minimizes input lag by synchronizing the CPU and the GPU for optimal responsiveness.

Powered by these three technologies, DLSS 3 enables upwards of 4x performance boosts, providing headroom for next-generation, path-traced rendering. 

DLSS Super Resolution has been available in Unreal Engine since 2021, making it easy to integrate NVIDIA AI scaling technology into Unreal Engine projects. NVIDIA has now released DLSS 3 for Unreal Engine 5.2, which includes Frame Generation and the latest NVIDIA Reflex version. For more information about Unreal Engine 5.1 and earlier, see step 2 in the installation guide later in this post.

To make integrating NVIDIA technology into your project as simple as possible, the new DLSS 3 Unreal Engine 5.2 package contains the Frame Generation, Super Resolution, and NVIDIA Reflex plugins all in a single download.

DLSS 3 technologies

The DLSS Frame Generation plugin uses Frame Generation to create entirely new frames by analyzing sequential frames and motion data from the Optical Flow Accelerator in GeForce RTX 40 Series GPUs.

Bundled inside the DLSS Frame Generation plugin is NVIDIA Reflex. Paired with DLSS 3, NVIDIA Reflex reduces onscreen latency by up to 2x compared to native rendering.

The DLSS Super Resolution plugin supports a variety of image quality modes—from Ultra Performance to Quality—determined by the native resolution relative to the DLSS output resolution. DLSS Super Resolution is customizable based on the needs of your game, with additional NVIDIA technologies included in the plugin:

• Deep Learning Anti-Aliasing Mode (DLAA) offers an AI-based anti-aliasing mode for users who have spare GPU headroom and want higher levels of image quality.
• NVIDIA Image Scaling is an open-source spatial upscaler and sharpening algorithm that is available for all platforms. 

The DLSS 3 Unreal Engine 5.2 plugin is delivered with the latest optimizations to NVIDIA AI algorithms, always learning and evolving with over-the-air updates.

How to install DLSS 3 for Unreal Engine

Follow these steps to download and install DLSS 3 for your Unreal Engine project.

1. Agree to the Terms of the License Agreement and download DLSS 3 for your version of Unreal Engine.
2. Unzip the DLSS folder. Only the 5.2 version of DLSS contains the Streamline/Frame Generation plugin.
3. Copy the plugin folders to install to the /Engine/Plugins/MarketPlace folder of your Unreal Engine directory. If you don’t currently have a /MarketPlace folder, create one.
4. Launch Unreal Editor, go to Plugins, and search for the plugins to activate. Search for “NVIDIA” to quickly list all of the included DLSS 3 plugins.
5. Activate and restart Unreal Editor.
6. Load the DLSS 3 Test project from the /Samples folder of the downloaded DLSS plugin file.

For prior versions of Unreal, you must build from the source and modify your source code with a small patch. For more information, see the included DLSS Frame Generation Quick Start Guide PDF in the download .zip file.

Tips for using DLSS 3 in Unreal Engine

After DLSS 3 is installed, follow these steps to verify that the Frame Generation, Super Resolution, and Reflex plugins are integrated into your project correctly.

• To confirm that DLSS Frame Generation is working, along with real-time statistics, navigate to project settings, and then to your preferences for the NVIDIA Streamline plugin. Toggle the Load Debug Overlay option.
  • The Load Debug Overlay option for Frame Generation works in the editor and can appear in development or debug builds, but won’t appear in production builds.
• To update Streamline automatically as well as DLSS AI algorithms with the latest improvements, use the same settings window to ensure that the Allow OTA Update option is enabled.
• In the Unreal Editor, Frame Generation only works from a new editor window (PIE) or in Standalone mode. It doesn’t work from the selected viewport or while editing.
• If any of the included DLSS 3 technologies aren’t working, check the output log or look for onscreen warning messages. A common issue may be that the NVIDIA drivers may have to be updated, for example.
• The DLSS 3 Unreal Engine plugin contains the latest NVIDIA Reflex technology, a newer version than the version currently built into Unreal Engine. While it’s possible to keep the earlier plugin enabled, and even use the earlier NVIDIA Reflex Blueprint scripts, we recommended that you disable the earlier NVIDIA Reflex plugin and use the new version bundled in DLSS 3 Streamline instead.
• We recommend that you set up all NVIDIA plugins through Blueprint scripts, as this enables you to conveniently activate plugins from menus and set preferences for users. However, if you need access to the console commands, they can be found under r.ngx. For more information about using console commands, see the DLSS Quick Start Guide PDF included in the DLSS 3 plugin download.
• When Frame Generation is on, we recommend that you disable VSYNC in your application. The DLSS 3 plugin can set VSYNC to behave incorrectly when active. VSYNC can be disabled with the r.vsync 0 console command.

Download DLSS 3 for Unreal Engine

DLSS 3 for Unreal Engine makes the latest NVIDIA advancements in neural rendering and performance multiplication easy to integrate into your UE project. Get started with the Frame Generation, Super Resolution, and Reflex plugins now.

DLSS 3 for Unreal Engine 5.2 is now available.

For more information, see NVIDIA technologies supported by Unreal Engine 5.