Meet the electric vehicle that’s truly future-proof. Electric-automaker NIO took the wraps off its fifth mass-production model, the ET5, during NIO Day 2021 last week. The mid-size sedan borrows from its luxury and performance predecessors for an intelligent vehicle that’s as agile as it is comfortable. Its AI features are powered by the NIO Adam Read article >
Imagine picking out a brand new car — only to find a chip in the paint, rip in the seat fabric or mark in the glass. AI can help prevent such moments of disappointment for manufacturers and potential buyers. Mariner, an NVIDIA Metropolis partner based in Charlotte, North Carolina, offers an AI-enabled video analytics system Read article >
Hey guys, I know this may not be the perfect place, but I though some of you may have the skills and the interest to apply to some recent job openings. I you are not interested in these jobs just ignore them, downvote them, but there may be other who not only find them useful but they can make a difference if you are interested in them apply directly to the link, we are searching for North America based individuals (for tax reasons)only, thanks!
Here in this example: Using pre-trained word embeddings | Keras, we can see that by providing pre-trained word embedding in embedding layer initialization, we can boost the performance of the model. But before doing that, they are removing the tokens which are not available in the current data-set. But I wonder, if it is helpful or not. If we kept all the tokens wouldn’t it be more helpful to classify text? As in that case the unknown words would also get some representations and which would be helpful in further classification process. Why we are not taking this advantage? Correct me if I am mistaken.
Blue noise textures are used in a variety of real time–rendering techniques to hide noise in a perceptually pleasing way
Blue noise textures are useful for providing per-pixel random values to make noise patterns in renderings. Blue noise textures are harder to see and easier to remove than noise made by either random number generators or hashes, both being white noise. To use a blue noise texture, you tile it across the screen, read the texture with nearest neighbor point sampling, and use that as your random value.
In this post, we add the time axis to blue-noise textures, giving each frame high-quality spatial blue noise and making each pixel be blue over time. This provides better convergence and temporal stability over other blue-noise animation methods. We also show you how to make non-uniform blue noise textures to allow for importance sampling. We go on a deeper technical dive in the follow up post, Rendering in Real Time with Spatiotemporal Blue Noise Textures, Part 2.
While othermethods combine blue noise and better convergence, they focus on convergence first and blue noise second. Our work focuses on blue noise first and convergence second, which makes for better renders at the lowest of sample counts, where blue noise has the most benefit.
A notable limitation to blue noise textures is that they work best in low-sample-count, low-dimension algorithms. For high sample counts, or high dimensions found in algorithms like path tracing, you would likely want to switch to low-discrepancy sequences to remove the error, instead of trying to hide it with blue noise.
Also worth mentioning is that that pixels under motion under TAA lose temporal benefits and our noise then functions as purely spatial blue noise. Pixels that are still even for a moment gain temporal stability and lower error, however, which is then carried around by TAA when they are in motion again. In these situations, our noise does no worse than spatial blue noise, so it should always be used instead, to gain benefits where available and do no worse otherwise.
Figure 1 shows an example of using blue noise compared to spatiotemporal blue noise.
Figure 1 uses stochastic single scattering, where free-flight distances are sampled using a series of blue noise masks over time. Traditional 2D blue noise masks (far left) are easy to filter spatially, but exhibit a white noise signal over time, making the underlying signal difficult to filter temporally.
Our spatiotemporal blue noise (STBN) masks (right of large image) additionally exhibit blue noise in the temporal dimension, resulting in a signal that is easier to filter over time. On the far right, we show two crops of the main image, as well as their corresponding discrete Fourier transforms over both space (DFT(XY)) and time (DFT(ZY)). The Z axis is time. The ground truth is shown in the insets in the large image (upper and lower right corners).
Scalars
Scalar spatiotemporal blue noise textures store a scalar value per pixel and are useful for rendering algorithms that want a random scalar value per pixel, such as stochastic transparency. You generate these textures by running the void and cluster algorithm in 3D but modify the energy function.
When calculating the energy between two pixels, you only return the energy value if the two pixels are from the same texture slice (have the same z value) or if they are the same pixel at different points in time (have the same xy value); otherwise, it returns zero. The result is N textures, which are perfectly blue over space, but each pixel individually is also blue over the z axis (time). In these textures, the (x,y) planes are spatial dimensions that correspond to screen pixels, and the z axis is the dimension of time. You advance one step down the z dimension each frame.
Figure 2 shows example textures and the XY and XZ DFTs for the spatiotemporal blue noise, an array of independent 2D blue noise textures, and 3D blue noise. Only our noise is blue over space (XY) and blue over time (Z), as you can see by the darkening of the center, where the low frequencies are attenuated.
In Figure 2, individual slices of 3D blue noise are not good over space, nor over time. Only our spatiotemporal blue noise is blue over both space and time. For more information about why 3D blue noise is not useful for animating blue noise, see Christoph Peters’ nice explanation in The problem with 3D blue noise.
Figure 3 shows a texture that is 50% transparent over a black background, using noise to do a binary alpha test (stochastic transparency) and filtered with temporal anti-aliasing (TAA).
Independent blue noise textures are a significant improvement over white noise by having an evenly spaced set of surviving pixels each frame. This is better for neighborhood sampling rejection, compared to white noise, which has clumps and voids of surviving pixels. Spatiotemporal blue noise does even better by making each pixel survive on frames that are evenly spaced temporally as well, making for a more converged, and more temporally stable result.
In Figure 3, white noise (left) is very noisy due to clumps and voids that aren’t present in blue noise (center). Our spatiotemporal blue noise does better by having pixels survive evenly not just over space but also over time.
Vectors
Vector spatiotemporal blue noise textures store a vector value per pixel and are useful for rendering algorithms that want a random vector per pixel, such as ray traced ambient occlusion. You generate these textures by running the algorithm from Blue-noise Dithered Sampling (BNDS) in 3D. You make the same modification to the energy function in that paper as you did for scalars in void and cluster. You only return a nonzero energy if they are from the same texture slice, or if they are the same pixel at different points in time.
The result is again N textures, which are perfectly blue over space, but each pixel individually is also blue over the z axis. Unit vectors can be used, which are useful for situations where you need direction vectors, and nonunit vectors can be used, which are useful when you just need an N dimensional random number, such as a point in space.
Figure 4 shows slices of vector valued spatiotemporal blue noise, as well as their frequency components over the space and time axis. This shows that they are blue over space and blue over time.
Vec1
Unit Vec1
Vec2
Unit Vec2
Vec3
Unit Vec3
XY[0]
DFT(XY)
DFT(XZ)
Figure 4. 128x128x64 Spatiotemporal blue noise textures and their frequencies shown for unit and nonunit vectors of dimension 1, 2, and 3.
Figure 5 shows 4 sample per pixel ray traced ambient occlusion (AO) using various types of unit vec3 noise. If a vector is facing towards the normal, it is negated. The difference in quality is apparent between white noise, independent blue noise textures, and spatiotemporal blue noise.
Importance sampling
The BNDS algorithm starts with a set of white noise textures and repeatedly swaps pixels at random, if the swap improves the energy function. There is no reason why these textures must be initialized to uniform white noise vectors, though.
When initializing them to a non-uniform distribution, the algorithm still works in creating blue noise textures. The result is spatiotemporal blue noise textures, which also happen to have a non-uniform histogram, which allows importance sampling. As you need the PDF per pixel to do importance sampling, you can either store the PDF(x) in the alpha channel or calculate the PDF from the value in the texture, such as by doing a dot product if it is cosine hemisphere-weighted or dividing by a normalization value passed in as a shader constant.
Figure 6 shows importance sampled, vector-valued, spatiotemporal blue noise textures.
Texture[0]
DFT(XY)
DFT(ZY)
Importance Map
Cosine Weighted Hemisphere Unit Vec3
N/A
HDR Skybox Importance Sampled Unit Vec3
Figure 6. Slices of importance sampled spatiotemporal blue noise, their DFTs, and the source image they are importance sampling. The alpha channel of the textures stores the PDF as a percentage between the minimum and maximum PDF.
Figure 7 shows four-sample per pixel ray traced AO again but using cosine-weighted–hemisphere, importance-sampled unit vectors. White noise makes a unit vec3, adds it to the normal, and normalizes. Blue noise and STBN have cosine-weighted hemispherical vectors stored in their textures, which are transformed into tangent space using a TBN basis matrix.
Looking at the ovals at the top of the carousel shows how blue noise does better than white noise.
Looking at the window frames in the upper right, you can see how STBN has less noise in the shadows than independent blue noise textures do.
In Figure 7, the difference between the noise is less obvious than when uniform sampling but is still there. The noise in blue noise is harder to see and easier to filter than white noise. The noise in STBN is the same way but is also lower magnitude.
Conclusion
Blue noise can be a great way to get better-looking images at low sample counts, like those found in real-time rendering. Blue noise can be useful in nearly any situation that needs one or more random values per pixel.
Have ideas for using blue noise? Download the textures, give it a try, and share your results in the comments. We’d love to see them!
Spatiotemporal blue noise textures add the time axis, providing better convergence of blue noise, without loss of quality of the blue noise error patterns.
In the first post, Rendering in Real Time with Spatiotemporal Blue Noise, Part 1, we introduced the time axis and importance sampling to blue noise textures. In this post, we take a deeper look, show a few extensions, and explain some best practices.
Neighboring pixels in blue noise textures have very different values from each other, including wrap around neighbors, as if the texture were tiled. The assumption is that when you have a function that renders a pixel , small changes in x result in small changes in y, and that big changes in x result in big changes in y.
When you put very different neighbor values in for x, you also then get very different neighbor values out for y, tending to make the rendered result have blue noise error patterns. This assumption usually holds, unless your pixel rendering function is a hash function, on geometry edges, or on shading discontinuities.
It’s also worth noting that each pixel in spatiotemporal blue noise is a progressive blue noise sequence over time but is progressive from any point in the sequence. This can be seen when looking at the DFT and remembering that the Fourier transform assumes infinite repetition of the sequence being transformed.
There are no seams in the blue noise that would distort the frequency content. This means that when using TAA, where each pixel throws out its history on different timelines, every pixel is immediately on a good, progressive sampling sequence when rejecting history, instead of being on a less good sequence until the sequence restarts, as is common in other sampling strategies. In this way, each pixel in spatiotemporal blue noise is toroidally progressive.
It is worth mentioning is that that pixels under motion under TAA lose temporal benefits and our noise then functions as purely spatial blue noise. Pixels that are still even for a moment gain temporal stability and lower error, however, which is then carried around by TAA when they are in motion again. In these situations, our noise does no worse than spatial blue noise, so should always be used instead, to gain benefits where available and do no worse otherlwise.
Denoising blue noise
Blue noise is more easily removed from an image than white noise due to digital signal processing reasons. White noise has randomization in all frequencies, while blue noise has randomization only in high frequencies.
A blur, such as a box filter or a Gaussian blur, is a low-pass filter, which that means it removes high frequencies but enables low frequencies to remain.
When white noise is blurred, it turns into noisy blobs, due to lower frequency randomization surviving the low-pass filter.
When blue noise is blurred, the high frequency noise goes away and leaves the lower frequencies of the image intact.
Blue noise uses Gaussian energy functions during its creation so it is optimized to be removed by Gaussian blurs. If you blur blue noise with a Gaussian function and are seeing noisy blobs remain, that means you must use a larger sigma in the blur, due to the lower frequencies of the blue noise passing through the filter. There may be a balance between removing those blobs and preserving more detail in the denoised image. It depends on your preferences.
Figures 2, 3, and 4 show how blue noise compares to white noise both when used raw, as well as when denoised.
In Figure 3, both images have only eight colors total, as they are only 1-bit per color channel. There are many more recognizable and finer details in the blue-noise–dithered image!
To see the reason why blue noise denoises so much better than white noise, look at them in frequency space. You apply a Gaussian blur through convolution, which is the same as a pixel-wise multiplication in frequency space.
If you multiply the blue noise frequencies by the Gaussian kernel frequencies, there will be nothing left, and it will be all black; the Gaussian blur removes the blue noise.
If you multiply the white noise frequencies by the Gaussian kernel frequencies, you end up with something in the shape of the Gaussian kernel frequencies (low frequencies), but they are randomized. These are the blobs left over after blurring white noise.
Figure 5 shows the frequency magnitudes of blue noise, white noise, and a Gaussian blur kernel.
Tiling blue noise
Blue noise tiles well due to not having any larger scale (lower frequency) content. Figure 6 shows how this is true but also shows that blue noise tiling gets more obvious at lower resolutions. This is important because blue noise textures are most commonly tiled across the screen and used in screen space. If you notice tiling when using blue noise, you should try a larger resolution blue noise texture.
Getting more than one value per pixel
There may be times you want more than one spatiotemporal blue noise value per pixel, like when rendering multiple samples per pixel.
One way to do this is to read the texture at some fixed offset. For instance, if you read the first value at (pixelX, pixelY) % textureSize, you might read the second value at (pixelX+5, pixelY+7) % textureSize. This essentially gives you an uncorrelated spatiotemporal blue noise value, just as if you had a second spatiotemporal blue noise texture you were reading from.
The reason this works is because blue noise textures have correlation only over short distances. At long distances, the values are uncorrelated, as shown in Figure 7.
Ideally if you want N spatiotemporal blue noise values, you should read the blue noise texture at N offsets that are maximally spaced from each other. A good way to do this is to have a progressive low-discrepancy sequence into which you plug the random number index, and it gives you an offset at which to read the texture.
We have had great success using Martin Robert’s R2 sequence to plug in an index, get a 2D vector out in [0,1), and multiply by the blue noise texture size to get the offset at which to read.
There is another way to get multiple values per pixel though, by adding a rank 1 lattice to each pixel. When done this way, it’s similar to Cranley-Patterson rotation on the lattice but using blue noise instead of white noise.
For scalar blue noise, we’ve had good results using the golden ratio or square root of two.
For non-unit vec2 blue noise, we’ve had good results using Martin Robert’s R2 sequence.
This method sometimes converges better than spatiotemporal blue noise but has a more erratic error graph, making it less temporally stable, and damages the blue noise frequency spectrum. Figure 8 shows the frequency damage and Figure 8 shows some convergence behavior. For more information about convergence characteristics, see the simple function convergence section later in this post.
Figure 9 shows a graph comparing real vector spatiotemporal blue noise to the R2 low discrepancy sequence that uses a single vector blue noise texture for Cransley-Patterson rotation.
These two methods are the way that others have animated blue noise previously. Either the blue noise texture is offset each frame, which makes it blue noise over space and white noise over time, or a low-discrepancy sequence is seeded with blue noise values, making it be damaged blue noise over space, but a good converging sequence over time.
Making vector-valued spatiotemporal blue noise through curve inversion
If you have a scalar spatiotemporal blue noise texture, you can put it through an inverted Morton or Hilbert curve to make it into a vector-valued spatiotemporal blue noise texture. We’ve had better results with Hilbert curves. While these textures don’t perform as well as the other methods of making spatiotemporal blue noise, it is much faster and can even be done in real time (Figure 10).
An interesting thing about this method is that we’ve found it works well with all sorts of dither masks or other scalar-valued (grayscale) noise patterns: Bayer matrices, Interleaved Gradient Noise, and even stylized noise patterns.
In all these cases, you get vectors that, when used in rendering, result in error patterns that take the properties and looks of the source texture. This can be fun for stylized noise rendering, but also means that in the future, if other scalar sampling masks are discovered, this method can likely be used to turn them into vector-valued masks with the same properties.
Stratification
The energy function of vector valued spatiotemporal blue noise can be modified to return nonzero only if the following conditions are true:
The pixels are from the same slice (same z value)
The temporal histograms of the pixels involved in the swap don’t get worse
If you do this, you end up with noise that is blue over space but stratified over time; the stratification order is randomized. Because stratification isn’t progressive, it doesn’t converge well until all samples have been taken but does well at that point (Figure 11).
Higher dimensional blue noise
The algorithms for generating spatiotemporal blue noise aren’t limited to working in 3D. The algorithms can be trivially modified to make higher dimensional blue noise of various kinds, although so far, we haven’t been able to find usage cases for them. If spatiotemporal blue noise is 2Dx1D because it is 2D blue noise on XY and 1D blue noise on Z, Figure 12 shows the frequency magnitudes of 4D blue noise, which are 2Dx2D and 2Dx1Dx1D, respectively, with dimensions of 64x64x16x16.
Point sets
Blue noise textures made with the void and cluster algorithm can be thresholded to a percentage value. That many pixels survive the thresholding and they are blue-noise–distributed. Our scalar-valued spatiotemporal blue noise textures have the same property and result in spatiotemporal blue noise point sets. Figure 13 shows that with the thresholded points of a scalar spatiotemporal blue noise texture, as well as the frequency amplitudes of those thresholded points.
These point sets are such that the pixels each frame are distributed in a pleasing spatial blue noise way, but you also get a different set of points each frame. That means that you get more unique pixels over time compared to white noise or other animated blue noise methods. Figure 14 shows five frames of accumulated samples of an image, using the importance map as the per pixel blue noise threshold value. Our noise samples the most unique pixels the fastest, while also giving a nice blue noise pattern spatially.
Simple function convergence
In this section, we discuss the convergence of simple functions using common types of noise, under both Monte Carlo integration and exponential moving average to simulate TAA.
Scalar Monte Carlo integration
Figure 15 shows convergence rates under Monte Carlo integration, like what you would do when taking multiple samples per pixel. HB LDBN is from A Low-Discrepancy Sampler that Distributes Monte Carlo Errors as a Blue Noise in Screen Space. While low-discrepancy sequences can do better than STBN, it is more temporally stable and also makes perceptually good error by being blue noise spatially. STBN Offset 1/3 shows that if you start STBN from an arbitrary place in the sequence, that it still retains good convergence properties. This shows that STBN is toroidally progressive.
Scalar exponential moving average
Figure 16 shows convergence rates under exponential moving average. Exponential moving average linearly interpolates from the previous value to the next by a value of 0.1. This simulates TAA without reprojection or neighborhood sampling rejection.
Vec2 Monte Carlo integration
Figure 17 shows convergence rates under Monte Carlo integration. Scalar STBN uses the R2 texture offset method to read two scalar values for this 2D integration. Because of that, it outperforms Vector STBN in the step function, which is ultimately a 1D problem, and also in bilinear, which is ultimately an axis-aligned problem. The reason why importance sampling has any error at all is due to discretization of both the vectors and the PDF values.
Vec2 exponential moving average
Figure 18 shows convergence under EMA. Exponential moving average linearly interpolates from the previous value to the next by a value of 0.1. This simulates TAA without reprojection or neighborhood sampling rejection.
Conclusion
While blue noise sample points have seen advancements in recent years, blue noise textures seem to have been largely ignored for decades. As shown here, spatiotemporal blue noise textures have several desirable properties for real-time rendering where only low sample counts can be afforded: good spatial error patterns, better temporal stability, and convergence, and toroidal progressiveness, just to name a few.
We believe that these textures are just the beginning, as several other possibilities exist for improvements to sampling textures, whether they are blue-noise–based, hybrids, or something else entirely.
It is worth noting that there are other ways to get great results at the lowest of sample counts, though. For instance, NVIDIA RTXDI is meant for this situation as well but uses a different approach.
NVIDIA partnered with GeoComputing Group and Lenovo on a high-performance, secure, hybrid platform that enhances productivity for geoscientists.
Whether working remotely or in the office, geoscientists depend on fast access to large and complex datasets to be productive. Yet, up to 40 percent of their time is spent waiting for data to load, with additional time wasted waiting for geoscience applications using high-cost, legacy IT systems.
To improve productivity for geoscientists, GeoComputing Group, Lenovo, and NVIDIA partnered to create a Remote Interpretation and Visualization Appliance (RiVA). The high-performance computing platform was specifically created for subsurface workflows including seismic analysis and reservoir simulation.
Using RiVA, oil and gas enterprises are able to access their data between 50x to 100x faster and reduce model deployment times significantly.
RiVA provides a high-performance, low-latency, consolidated environment for remotely hosted, industry-standard applications used in exploration and production. The platform integrates Lenovo servers and storage with NVIDIA RTX, GPUs, NVIDIA RTX virtual workstations, and Infiniband high-speed networking.
It enables deployment of high-quality, 3D virtual workstations for large datasets and graphically intensive workflows in the oil and gas industry. A Remote Visualization Server (RVS) Module manages the hardware, software, and services for a comprehensive and performant scaling of virtual desktops for petro-technical workflows.
Geoscientists using RiVA can access datasets in a secure data center with a centralized compute model. Applications and data are secured in a private cloud, which helps increase application reliability, speed, and stability. Hybrid workers can overcome productivity drains caused by slow data delivery, delayed application responsiveness, or unexpected downtime.
Oil and gas companies using RiVA have reduced time to production for their infrastructure by up to 70 percent compared to traditional infrastructure builds. They have also experienced between a 100 and 400 percent return on investment in less than a year, providing a more efficient work experience for their geoscientists. With higher accuracy in subsurface analysis and accelerated workflows, enterprises are empowering geoscientists to find more oil in less time.
Explore NVIDIA energy solutions creating a more sustainable future. Learn more about the RiVA high-performance computing platform in the video below.
2021 saw massive growth in the demand for edge computing — driven by the pandemic, the need for more efficient business processes, as well as key advances in the Internet of Things, 5G and AI. In a study published by IBM in May, for example, 94 percent of surveyed executives said their organizations will implement Read article >
Hey guys, I know this may not be the perfect place, but I though some of you may have the skills and the interest to apply to some recent job openings. I you are not interested in these jobs just ignore them, downvote them, but there may be other who not only find them useful but they can make a difference if you are interested in them apply directly to the link, we are searching for North America based individuals (for tax reasons)only, thanks!