Brief Introduction

The main goal of this project is to apply non-photorealitic rendering techniques onto existing 2D images in order to achieve impressionist effects.

So What is Impressionism?

By definition on Wikipedia, impressionism is a 19th-century art movement characterized by relatively small, thin, yet visible brush strokes, open composition, emphasis on accurate depiction of light in its changing qualities (often accentuating the effects of the passage of time), ordinary subject matter, inclusion of movement as a crucial element of human perception and experience, and unusual visual angles.

You may not be fimiliar with the definition of impressionism but you must be pretty familiar with names of famous impressionists and their masterpieces.

This project is mainly inspired by the paper "Processing Images and Video for An Impressionist Effect" from Peter Litwinowicz.

First Attempt: Fixed Stokes

One notable techique used by impressionist is using small, thin, yet visible brush strokes. So in order to achieve that effect, I tried to draw short strokes with preset lengths, widths and orientations using colors at the center of strokes.

Here is the result:

Note that in this approach all stokes are generated in random order to help create a hand-touched look. However, the whole image is still way too unnatural compared to the aboving impressionist masterpieces. Impressionist artists are not robots so they probably cannot always paint stokes with the same length, width and orientation.

Add Some Randomness!

You will notice that stroke at the sky region in Impression, Sunrise by Monet has different colors and orientations. As I mentioned before, impressionist artists are trying to paint visible strokes. Therefore, since we are using the center pixel color at each strokes, strokes at a area with unnoticeable color difference will not be distinct enough from each other. A solution proposed by Peter Litwinowicz is to add some random perturbation to each individual stroke's parameters(i.e. length, width, orientation and color).

Here is the comparision of results:

The generated image looks way better than before. But we are still some distance away from impressionist effects.

Let There Be Edges

Now the strokes in our generated image looks somewhat similar to impresionist paintings. However, the image still looks blurred. Well, this is because the previous approach does not pay any attention to edges. Hence, strokes are clipped when it reaches an edge.

Here is result of edge detection:

Here is the comparision of results:

Note that the edge detector uses a low-pass(Gaussian filtered) image so that it can be less sensitive to some minor details.

What About Gradients?

The generated image still does not seem to be enough impressionist-ish. In previous projects, professor Hoiem mentioned that gradient domain processing can also be used to create non-photorealistic rendering effects. So it must have something to do with gradients.

Lets take a look at the Starry Night Over the Rhône by Van Gogh:

You may notice that the strokes are not fixed to around 45 degress to the x axis. So how are we going to determine the orientation of each stroke? The answer is "gradient".

In NPR(non-photorealistic rendering), we are trying to preserve edges and simultaneously minimize the gradient within areas without edges. Hence, we can use the gradient normal for the orientation of strokes since gradient direction represents the the direction which intensity changes most frequently.

Here is the comparision of results:

Here is another comparision of results:

Using gradient normals as orientation of strokes also helps preserving edges.

More About Gradients

In the preious results, the stroke orientation in the background looks kind of unnatural again. This problem is also related to image gradients.

Lets take the apple image as example:

You can see that the gradient directions in the background are almost all zero or near zero(Well that's because it has a white background). In that case, we cannot simply rely on them. Through observation, impressionist artists tend to use the gradient directions around edges for strokes far away from edges. Moreover, little difference in Gx and Gy can cause huge difference in angles.

Peter Litwinowicz also proposed a solution to this problem in his paper:

To accomplish this, gradient values are “thrown out” at pixel locations where the Magnitude((Gx,Gy)) is near zero. In this implementation, this is approximated by the test: |Gx| < 3.0 and |Gy| < 3.0, which was empirically found to be useful. The gradient at pixels with near zero gradient magnitude are then replaced by interpolating surrounding "good" gradients

At first I used a brute force seraching algorithm for finding the nearest non-zero neighbor for near-zero pixels but the running time is terrible when the resolution of input images went up. So I am using a kd-tree instead for better performence.

Here is the result of replacing "bad" gradients:

Here is the comparision of results:

More Results

Altgeld Hall(Higher Resolution Version)

Palace of Westminster

Some Landscape

Mona-Lisa

Sunset

Siebel Center

Further Discussion

There still remains a lot to be improved for this project:

• Code Speed

  It almost took my computer two hours to generate the Shanghai Night View image(1500 * 993). The Matlab profiler says the antialiasing code account for the majority of running time. I think optimizing that part of code should result in a great improvement of overall running time.

• Brush Stroke Texture

  If you take a closer look at Van Gogh or Monet's painting, you will find that the brush stroke textures of each stroke are varying from each other and the color within one stroke is also varying. This is probably a combinational effect of gravity, the strength when painting the stroke or other complicated factors. I believe it will be difficult to completely replicate impressionist stroke textures but it certainly should be better than using uniform color line-shaped stokes.

• Using Bilinear Interpolation for Stroke Colors and Gradient Directions.

  I used the values from center pixels of each stroke and achieved reasonably good quality. However, Peter Litwinowicz suggested to use bilinear interpolation of colors and gradient directions. I think this can slightly improve the color and orientation accuracy for strokes. For example, if there is a pixel with very distinct color in your input image and, with extremely bad luck, the god of randomness decides to use its color and gradient normal at that position in your output image, it may result in a very werid-looking and unnatural stroke.