Denoising with Kernel Prediction and
Asymmetric Loss Functions

Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, Jan Novák

Denoising

Monte Carlo path tracing

The images we want to denoise come from a path tracer. To render a image with a path tracer requires simulating the physics of light, and is very expensive. A single production frame can take up to a day to render.

For each pixel, the algorithm traces many light paths and averages their contributions. After a few light paths, you get a noisy estimate of the image, and the magnitude of this noise decreases very slowly. Notice the exponential scale here. In the beginning, the images improve rapidly, but the tail is long.

This is where denoising comes in. We want to cut off the tail. We use a clever filter to take a noisy image, and filter is so that it looks like a converged one.

This is a challenging task, but fortunately we can get all kinds of additional information from the renderer to help us. Examples here are surface normals and texture maps.

In our case, we take a supervised machine learning approach. Our denoiser is a CNN that we train with a dataset of lq/hq pairs.

Input

Denoised

Kernel Prediction

Basic single-frame denoiser (Bako et al. 2017)

Kernel

Image

Color

Features

Multi-renderer support source-aware encoders
Temporal stability cross-frame denoising
Denoising of low frequencies a multi-scale approach
Adaptive sampling the error predictor
User control asymmetric loss functions

When we showed the denoiser to our artists last year, they got very enthusiastic,

but they also pointed out a few areas in which it could be improved. This year's paper explains how we designed a number of extensions to the basic kernel prediction architecture.

One challenge we faced, is that we have many production studios that all generate noisy images with slightly different characteristics. We introduce source-aware encoders to help us deal with this variety.

Also, denoising frames one by one does not give great results for animated sequences. I will tell you about how we denoise sequence to ensure temporal stability.

On a similar line, we introduced multi-scale denoising to our denoise low frequencies in a more efficient way.

and after denoising an image, we can now predict its error, so we can instruct the renderer in which areas it would be helpful to spend more computation.

Near the end of the talk, I will tell you about asymmetric loss functions, and how they give our users run-time control over the strength of the denoiser.

Features

Multi-renderer support source-aware encoders
Temporal stability cross-frame denoising
Denoising of low frequencies a multi-scale approach
Adaptive sampling the error predictor
User control asymmetric loss functions

Multi-renderer support

Baseline generalization ability

Renderer B Input

Trained on Renderer A data

Retrained on Renderer B data

Scene by nacimus (Blendswap)

Multi-renderer support

Source-aware encoders

Multi-renderer support

Data efficiency of encoder training

Multi-renderer support source-aware encoders
Temporal stability cross-frame denoising
Denoising of low frequencies a multi-scale approach
Adaptive sampling the error predictor
User control asymmetric loss functions

Noisy input (32 SPP)

Denoised (still frames)

Denoised (7 frame temporal window)

Temporal denoising

Reusing a pre-trained single-frame denoiser

(Chaitanya et al. 2017)

To achieve this, we take the neural network we had before, but drop the kernel prediction part, and just keep it as a feature extractor.

And we don't just do this on a single frame, but we do it independently for each frame in the temporal neighborhood.

We use motion vectors to align these features over time, and train a temporal combiner, that takes these extracted features, and predicts weighting kernels again. This time, they are not just for the center frame, but there are weights for each of the frames in the neighborhood.

We can sum up the contributions from each frame to get a denoised image.

[Click] There was a paper last year by Chaitanya and colleagues where they denoise sequences interactively. The main difference with this approach, is that in the interactive setting, they only have access to past frames, which they leverage using a recurrent architecture. Extending this approach to make use of future frames as well would require a bidirectional recurrent network, and we value the simplicitly of this design. Please see the paper for a quality comparison in comparable scenarios.

Multi-renderer support source-aware encoders
Temporal stability cross-frame denoising
Denoising of low frequencies a multi-scale approach
Adaptive sampling the error predictor
User control asymmetric loss functions

Multi-scale denoising

Cheaper denoising of low frequencies

Multi-scale denoising

Multi-renderer support source-aware encoders
Temporal stability cross-frame denoising
Denoising of low frequencies a multi-scale approach
Adaptive sampling the error predictor
User control asymmetric loss functions

Adaptive sampling

with an error predictor

Predicted sampling map

Multi-renderer support source-aware encoders
Temporal stability cross-frame denoising
Denoising of low frequencies a multi-scale approach
Adaptive sampling the error predictor
User control asymmetric loss functions

Summing up …

Multi-renderer support source-aware encoders
Temporal stability cross-frame denoising
Denoising of low frequencies a multi-scale approach
Adaptive sampling the error predictor
User control asymmetric loss functions

We presented a method for training with multiple different datasets, allowing data-efficient repurposing of a trained model to a new renderer.

Then we disussed how to reduce flickering, both spatially and temporally,

And we talked about predicting denoising errors with a neural network to improve adaptive sampling.

Finally, we presented Asymmetric loss fucntions, which give artists run-time control over denoising strength. We hope this idea can also be useful in other applications!

We skipped over lots of details, of course. For example, I haven't said anything about how many layers there are in each module, or what the kernel-sizes are, or even the particular network architecture. This is on purpose: the paper implements a succesful denoiser, but all the ideas presented today would also work in a denoiser that is layed out slightly differently.

Denoising with Kernel Prediction andAsymmetric Loss Functions

Denoising

Input

Denoised

Kernel Prediction

Features

Features

Multi-renderer support

Multi-renderer support

Multi-renderer support

Noisy input (32 SPP)

Denoised (still frames)

Denoised (7 frame temporal window)

Temporal denoising

Multi-scale denoising

Multi-scale denoising

Multi-scale denoising

Adaptive sampling

Predicted sampling map

Summing up …

Input

Denoised

Input

Denoised

Input

Denoised

Input

Denoised

Denoising with Kernel Prediction and
Asymmetric Loss Functions