U NIVERSITY OF C ALIFORNIA , B ERKELEY
CS294-26: Image Manipulation and
Computational Photography
Final Project Report
Ling-Qi Yan
December 14, 2015
1 I NTRODUCTION
Sandpainting is the art of pouring colored sands, and powdered pigments from minerals or
crystals, or pigments from other natural or synthetic sources onto a surface to make a fixed,
or unfixed sand painting.
Figure 1: Illustration of sand painting. (Left) Before painting. (Middle) After painting. (Right)
Colored sand used for painting.
However, generating sand paintings automatically using computers still remain untouched.
The main reason is that, sand paintings gives us an apparent granular appearance, which is
1
well known as high frequency. State of the art image processing methods, including computer
vision and machine learning, are mostly based on low frequency hypothesis, thus cannot be
easily adopted for sand painting purpose.
On the other hand, in the rendering research domain, [4] proposed a method to rendering
glints from high frequency normal mapped surfaces, which is promising in generating realistic glints so that the sand paintings look more convincing. However, their method works
only for direct reflections i.e. single scattering without considering interactions among sand
crystals, while in reality multiple scattering within the sand volume dominates. So it is also
not suitable for our need.
In this report, we propose a novel method to generate sand paintings. Our method is purely
image-based. Given a input photograph with different regions, we refer to a random process
to generate sand crystals for each region. Note that, our method has nothing to do with image quilting. The sand is actually generated from the input image rather than copied from
anywhere else.
2 I MAGE B ASED S AND PAINTING
2.1 M OTIVATION AND BACKGROUND
Figure 2: Steps of sand painting. (Left) Step 1: Pickle out the sticker with a toothpick. (Right)
Step 2: Lightly sprinkle with right colored sand.
Our work is motivated by the actual process of modern sand painting. The customers are
given the outline of the painting already, dividing the paint into regions. For each region, it is
glued and covered with a sticker. When painting, there are three main steps:
1. Pickle out the sticker with a toothpick.
2. Lightly sprinkle with right colored sand, and smear evenly with finger or tools.
2
3. Shake off the rest of the sand.
Continue for other regions with different colors, a sand painting is finished.
2.2 O UR METHOD
We incorporate the above mentioned physical steps into our method. Our method slightly
modifies them and consists of the following steps:
1. Perform image segmentation.
2. Simulate the process of sprinkling sand.
3. Extract boundaries.
4. Sand paint boundaries.
We now look at each individual step in detail.
2.2.1 I MAGE SEGMENTATION
Figure 3: An example of image segmentation. (Left) The input image. (Right) The output
image after image segmentation.
Given an input image, We first perform an image segmentation, thus we have different regions. This is a preparation step since real world images are certainly not well segmented like
the image given for sand painting.
We refer to the K -means algorithm [2], which is an interactive technique that is used to partition an image into K clusters. It works by randomly selecting K cluster centers, then assign
each pixel in the image to a cluster that minimizes the distance between the pixel and the
cluster center. By distance, here it means the difference of color and location between a pair
of pixels.
3
The image segmentation step gives us the index of region it belongs to for each pixel. We use
this to find the average color of each region.
Fig. 3 shows an example of image segmentation. As it is shown, after the segmentation, the
images are partitioned into several regions with constant color.
2.2.2 S AND SPRINKLING
Figure 4: An example of sand sprinkling. (Left) The input image. (Right) The output image
after image segmentation and sand sprinkling.
Since image segmentation tells us how the input image is partitioned into regions, we’re now
able to sand paint each region.
Think about the physical process. When we pour sand onto a sticky region. Some of the sand
crystals are stuck, while some others bounced and stuck somewhere else. If we focus on a
specific location (i.e. one pixel in the output image), when we throw a sand crystal right at it,
it might not end up being there exactly. Thus, we model the probability of staying at different
locations using a Gaussian distribution as
p(x + i , y + j ) = G(i ; σp ) · G( j ; σp )
(1)
where (x, y) is the pixel we want the sand crystal to land, (x + i , y + j ) is the actual location it
finally lands. Furthermore, using Gaussian distribution is reasonable, since the sand crystal
always tend to land close to where it should be, but could result in an outlier as well.
Given the distribution, we now perform a random process to simulate sand sprinkling for
each pixel. We randomly pick several sand crystals, perturb its location using the given Gaussian distribution. This could be done using any Gaussian random number generator such as
Box-Muller transform.
Once the location of a sand crystal is decided, we need to figure out its color. However, a sand
appears quite different looking from different directions. Instead of referring to rendering actual appearance, we seek a simple solution but effective and accurate. To do that, we analyze
4
an actual sand painting, finding how the colors distribute within regions where constant colors are expected in its original image. Once we know this distribution, we can directly apply
it to choose the color for any sand crystal, given its corresponding averaged color from the
image segmentation step.
Figure 5: Illustration of analyzing color distributions. (Left) An actual sand painting. (Middle)
A patch extracted from the painting, which is supposed to have constant color if it
were not sand painted. (Right) The histogram of the patch, note its shape, mean and
standard deviation.
Fig. 5 illustrates the idea. We take a patch which is supposed to have constant color from
an actual sand painting. Then we plot the histogram of its grayscale image. We find that the
color distribution looks exactly like a GGX distribution [3] with standard deviation σc ≈ 0.3.
Thus, the color of a sand crystal could be written as
c(c 0 ) = X (c 0 ; σc )
(2)
where c 0 is the averaged color of the segment containing the pixel to which the sand crystal
is thrown.
However, a GGX distribution doesn’t give us enough glints to match the real appearance of
sand paints. So, we manually enhance a random set of pixels’ intensities to simulate a glinty
appearance.
2.2.3 E XTRACT BOUNDARIES
As Fig. 1 shows, the sand paintings often have clear boundaries given by the manufacturer.
Thus, to make the sand painting look more realistic, we need to sand paint the boundaries as
well.
We refer to a simple Sobel edge detector by extracting horizontal and vertical boundaries, using the magnitude of gradient as the strength of edges. Fig. 6 shows the boundaries extracted.
Note that, this step is optional, especially if the input images given already have clear and
thick boundaries.
5
Figure 6: An example of extracting boundaries. (Left) The input image. (Right) Sobel boundaries.
2.2.4 S AND PAINT BOUNDARIES
Once the boundaries are extracted, we perform similar sand sprinkling, but with different
transparency according to the strength of edges, rather than using colors.
Note again this step is optional. We can also directly mark the boundaries without sand painting it.
3 R ESULTS
Here we show a couple of results generated using our method, together with the original images for comparison.
As we can see in Fig. 7 and Fig. 8, our method generates visually convincing sand paintings.
The platform is a 2013 Macbook Pro laptop with a 2.4 GHz Intel Core i7 processor. The time to
generate a 480P sand painting takes no longer than 1 minute with a Python implementation
using only one core.
4 C ONCLUSION AND F UTURE W ORK
In conclusion, we’ve presented an image based method to accurately generate sand paintings. Our method is physically based, requiring no training or extra data. And it generates
sand paintings in real time. We show comparisons with original input images to show that
our sand paintings are visually convincing and are far from merely noise.
In the future, we would like to cover more detailed process, such as smearing the sand so that
6
there’s no blank dots in the output image, and convolute each region with a filter to produce
a saturated appearance to simulate multiple scattering within the sand volume, etc. In fact,
the more we correctly model the actual physical process, the more realistic our sand paintings
will be. Another direction will be generating sand videos, which would be challenging since
the granular appearance must not change over frames. That way, a deterministic random
process [1] might help.
R EFERENCES
[1] J AKOB , W., H A Å AAN
˛ , M., Y AN , L.-Q., L AWRENCE , J., R AMAMOORTHI , R., AND
M ARSCHNER , S. Discrete stochastic microfacet models. ACM Transactions on Graphics
(Proceedings of SIGGRAPH 2014) 33, 4 (2014).
[2] K ASHANIPOUR , A., M ILANI , N. S., K ASHANIPOUR , A. R., AND E GHRARY, H. H. Robust
color classification using fuzzy rule-based particle swarm optimization. In Image and
Signal Processing, 2008. CISP’08. Congress on (2008), vol. 2, IEEE, pp. 110–114.
[3] WALTER , B., M ARSCHNER , S. R., L I , H., AND T ORRANCE , K. E. Microfacet models for
refraction through rough surfaces. In Proceedings of the 18th Eurographics conference on
Rendering Techniques (2007), Eurographics Association, pp. 195–206.
[4] YAN , L.-Q., H A Å AAN
˛ , M., J AKOB , W., L AWRENCE , J., M ARSCHNER , S., AND R AMAMOOR THI , R. Rendering glints on high-resolution normal-mapped specular surfaces. ACM
Transactions on Graphics (Proceedings of SIGGRAPH 2014) 33, 4 (2014).
7
Figure 7: Results part one. (Left column) The input images. (Right) The generated sand paintings.
8
Figure 8: Results part two. (Left column) The input images. (Right) The generated sand paintings.
9