As large language models (LLMs) continue to grow in size and complexity, the performance requirements for serving them quickly and cost-effectively continue to…
As large language models (LLMs) continue to grow in size and complexity, the performance requirements for serving them quickly and cost-effectively continue to grow. To deliver high LLM inference performance, an efficient parallel computing architecture and a flexible and highly-optimized software stack are required. Recently, NVIDIA Hopper GPUs running NVIDIA TensorRT-LLM inference software set…
Checkpoint and restore functionality for CUDA is exposed through a command-line utility called cuda-checkpoint. This utility can be used to transparently…
Edgeless Systems introduced Continuum AI, the first generative AI (GenAI) framework that keeps prompts encrypted at all times with confidential computing by…
Phi-3-Medium accelerates research with logic-rich features in both short (4K) and long (128K) context.
Trained on 600+ programming languages, StarCoder2-15B is now packaged as a NIM inference microservice available for free from the NVIDIA API catalog.
Featured in Nature, this post delves into how GPUs and other advanced technologies are meeting the computational challenges posed by AI.
Gemma 2, the next generation of Google Gemma models, is now optimized with TensorRT-LLM and packaged as NVIDIA NIM inference microservice.
Brev.dev is making it easier to develop AI solutions by leveraging software libraries, frameworks, and Jupyter Notebooks on the NVIDIA NGC catalog. You can use…