CLOUD CHARACTERIZATION USING LOCAL TEXTURE INFORMATION
Antti Isosalo, Markus Turtinen and Matti Pietikäinen
Machine Vision Group
Department of Electrical and Information Engineering
University of Oulu, Finland
{isosaloa,dillian,mkp}@ee.oulu.fi
ABSTRACT
In this paper, local texture information is used to classify
clouds in the sky views. The proposed system is designed
for practical meteorology using a camera directed to horizon. The goal is to study how local texture measures,
namely local binary patterns (LBP) and local edge patterns (LEP), can distinguish different cloud genres. The
approach indicated very good performance in a five class
classification experiment. The results were promising also
in a more difficult case where all the ten cloud genres and
clear sky were considered. As known, land-based cloud
classification resolves many problems which are present in
satellite images.
Fig. 1. An example of an all-sky image.
1. INTRODUCTION
Weather has become a topic of interest in unexpected levels
— and not the least because of the feared climate change.
The need for comprehensive and consistent cloud observations is highly important. Normally clouds are being detected by human observers, satellites, humidity soundings,
ceilometers and pyrometers.
Nevertheless of continuous observation of clouds and
sky, the automatic analysis and weather prediction is still
an open issue. This is reasonable because giving a description of the cloudy sky is a very difficult problem. From the
practical meteorological point of view, cloud altitude and
general cloud appearance combined to other indicators such
as air temperature and pressure are equally important to create reliable weather predictions. This makes automatic description of the weather more difficult. Despite the possible
challenges, there is a clear requirement for automatic cloud
classification.
Machine vision provides an interesting set of tools for
cloud analysis. Typical vision-based approaches use satellite images [1, 2]. Though functional systems exist, there
are some limitations related to this kind of approach that
lower the performance. The problems are mainly caused
by thick cloud layers that prevent the camera eye from seeing what is hidden behind the topmost (e.g. high or middle
clouds) opaque (non-see-through) layer. In this work, the
land-based imaging is used instead. The land-based cloud
classification faces similar problems caused by low or middle clouds, but separate clouds and cloud types can be more
easily distinguished from the earth surface.
Automatic cloud classification using land-based imaging has been a topic of some research. Peura et al. [3] used
all-sky imager data (see Fig. 1) using a wide-angle optics
and extracted features from the images using different convolution masks. They classified cloud images into predefined categories using k-NN classifier. Similarly, Buch and
Sun [4] used whole-sky imager (WSI) and tried to classify
each pixel in the image with a tree-based approach. They
used texture, position information and pixel brightness features in their system. In this paper, instead of using a wholesky imager, a camera directed to the horizon is used. The
camera was attached to a weather radar tower positioned
over a hill.
When observing the sky from a bit higher than
the earth surface, the only limiting factor is the meteorological visibility — not landscape or vegetation.
Traditional weather and cloud observations (see URL:
http://www.metoffice.gov.uk/publications/clouds) are made
specifically from earth surface, so this must be kept in
mind when using this kind of arrangement. With the kind
of images used in this research, vertical dimensions appear nicely when clouds are viewed from the side. This
opens more possibilities to detect Cumulonimbus clouds
— Cumulonimbus clouds are normally associated with
thunderstorms and showers of rain. When the camera is
directed to horizon we do not need to rely on darkness
to detect thick clouds, as it is affected by the amount and
the thickness of cloud layers above and overall cloudiness.
There are also less undesired reflections by default in this
imagery than the all-sky images sadly contain.
In this paper, cloud classification is considered as a texture characterization problem. Cloud texture appearance is
modelled with local texture measures using the local binary
pattern (LBP) methodology [5]. LBP is invariant against
any monotonic changes in the gray-scale and has proven
to be very robust in numerous real-world texture analysis
tasks. The robustness in varying illumination conditions
makes the LBP attractive for cloud texture classification as
well. Instead of being a pure texture statistic, LBP has
turned out to be more general visual descriptor [6]. It combines the microtextons and shape of texture elements in a
compact way. This is useful in cloud image analysis where
the shapes of the cloud are in important role.
In this paper, different versions of the LBP and its extensions are systematically experimented in the cloud classification application. In addition, the local edge patterns
(LEP) [7] based on the LBP are also experimented to confirm the hypotheses of the usefulness of the local texture
information in cloud classification.
2. PROPOSED APPROACH
The strength of texture is that it is based on instinctive impression of visual similarity. For one’s part, the feature extraction method and the metric used in comparing feature
vectors play a key role in achieving comparative classification results.
The local image features extracted with LBP or LEP
represent wide variety of different microtextons. Statistical
properties of such texture elements are able to encode huge
amount of information about the image content. In cloud
image analysis application LBP and LEP can be used to describe the general appearance of the cloud as well as the
shape and structures of different clouds. Typically different cloud genres can be visually distinguished using these
characteristics.
2.1. Local Binary Patterns
The local binary pattern (LBP) operator is a theoretically
simple, yet very powerful and gray-scale invariant method
of analyzing textures [5]. In practice, the LBP operator
combines characteristics of statistical and structural texture
analysis: it describes the texture with micro-primitives, often called textons, and their statistical placement rules. LBP
is very easy to calculate. For each pixel in an image, a binary code is produced by thresholding its neighborhood (8
pixels) with the value of the center pixel. A histogram is
then constructed to collect up the occurrences of different
binary patterns representing different types of curved edges,
spots, flat areas etc.
In this study, the most common variants of LBP were
used [6]. LBP-features are extracted from gray-scale images. With multi-scale LBPu2
8,1+16,2+24,3 there were need
for scaling the pictures as described in [8]. With multi-scale
LBPFu2
8,1+8,2+8,3 [9], where F stands for filtering, Gaussian
low-pass filtering was used. In the multi-scale approach the
resulting feature vector is concatenated from histograms
obtained by different radii and/or number of samples in
a circular neighborhood. For example in multi-resolution
LBPu2
8,1+16,2+24,3 the first operator uses a neighborhood
radius of one pixel and eight neighborhood samples, the
second operator uses two as the neighborhood radius, and
considers 16 neighborhood samples and the third operator
uses three as the neighborhood radius, and considers 24
neighborhood samples — u2 stands for uniform codes
(at most two one-to-zero or zero-to-one transitions in the
circular binary code) as presented in [5].
For classification with the LBP, a k-NN classifier with
the χ2 -distance measure was used.
2.2. Local Edge Patterns
The LEP is similar to the LBP, but it is calculated from the
edge images [7, 10]. The motivation of using LEP is its
robustness in translation, rotation and scale of textures. In
addition, it is known that the shapes of cloud edges provide valuable supporting information to both traditional and
automated cloud classification. These are automatically incorporated in the LEP features.
Fig. 2 shows how the LEP features are extracted. In
practice, the feature histogram has twice as many bins as
the LBP because the edge pixels have to be considered separately.
Before the LEP features can be calculated, some preprocessing is needed. The threshold to be used with LEP
in the edge image creation proposed by Yao and Chen [10]
was too high for our application. Linked to the same problem, some normalization was also needed to minimize the
effects brought by sunlight. Therefore a normalization procedure, say modified FCF-method, presented by Liao and
Chen [7] was utilized — it is based on FCF-method proposed by Finlayson et al. [11]. There was some loss of information with the normalization method, but within limits
of acceptance. Then the luminance component representing gray-scale information is obtained using the equation
a block from
LEP-mask
an edge image
1
1
1
1
2
1
1
0
128 256
8
0
0
1
64
16
32
4
multiplication
1
2
128 256
0
0
4
0
16
LEP-code for the center pixel in an edge image block:
1 + 2 + 4 + 16 + 128 + 256 = 407
eral times to increase statistical accuracy and reliability. Results are presented as confusion matrices and also by mean
classification rate, the rate of correctly classified samples,
related to the repetitions of the test.
In detail, the mean classification rate is the mean of all
calculated classification rates attained during the mentioned
iterations. Classification rate is the sum of diagonal values of the confusion matrix divided by the number of the
samples that are currently tried to classify. The results are
represented percentages.
The weakness of k-NN classifier concerning high dimensional feature vectors is noted, but the fight against the
curse of dimensionality was out of the scope of this research.
3. EXPERIMENTS
This section describes our test arrangements and provides
results obtained with different LBP and LEP operators.
Fig. 2. LEP feature extraction.
3.1. The Data Used
Yn = 0.299Rn + 0.587Gn + 0.114Bn , where Rn , Gn ,
and Bn are the normalized RGB components. Also an image specific (adaptive) threshold was used to produce the
binary image from which edges were detected using the Sobel method (which suits well for the task [10]).
Histograms consisting feature distributions obtained by
different versions of LEP, namely LEPSEG and LEPINV,
were calculated as described in [10]. In the actual experiments we decided to make concatenated versions of LEPSEG and LEPINV. It provided the best results in terms of
mean classification rate. Only the tails of LEPSEG and
LEPINV were used. The tail consists of bins that originate from those center pixels of the edge image block that
are edge pixels. The result is referred as LEP8,1 in Table 1.
The same lookup tables were used with LEPINV as with the
LBP to achieve codes that meet different conditions, here
rotation invariance.
Yao and Chen used histogram intersection as similarity
measure. We have used the χ2 -distance with k-NN, since
distance functions can be used to achieve the same goal. It
also helps us to perform comparison between the two methods, LBP and LEP.
The Finnish Meteorological Institute (FMI) has 10 (year
2006) similar cameras in active use providing sky images similar to the ones used in this work (see URL:
http://www.vimpeli.fi/Default.aspx?id=408363). The camera, Ikegami ICD-848P, is equipped with normal optics
and it employs a high-sensitivity CCD-sensor with about
440,000 pixels.
The original images (704 by 576 pixels and 16 million
colors) are stored in JPEG-format before they are transmitted to FMI’s server. An example image is shown in Fig.
3.
2.3. Classification
In the classification, each cloud texture sample was modelled with its own feature histogram. For actual classification, a k-NN classifier with χ2 -distance measure was used.
At this point, no other classifiers were utilized.
Classification was done by dividing the whole data into
separate training and testing sets. Tests were repeated sev-
Fig. 3. An example of the original images used in our experiments.
The camera is static and therefore it was very straightforward to crop the images in our disposal in a way that only
Table 1. The mean classification rates obtained with different LBP and LEP operators
Method
Bins 11 classes 5 classes
LPB8,1
256
49%
69%
LPBri
36
37%
64%
8,1
u2
LPB8,1
59
49%
70%
LPBriu2
10
38%
60%
8,1
LBPu2
857
49%
71%
8,1+16,2+24,3
LBPriu2
54
42%
65%
8,1+16,2+24,3
MS-LBPFu2
177
49%
68%
8,1+8,2+8,3
MS-LBPu2
2571
57%
80%
8,1+16,2+24,3
MS-LBPriu2
162
47%
76%
8,1+16,2+24,3
LEP8,1
292
32%
52%
Fig. 4. An example of the cropped input image and the samples created from it using the labels shown in the middle.
the effective area was left in the images, e.g. clouds and
sky. At this point, the resulting images were saved in such
a format that no more information is lost, since the original
images were only available on JPEG-format as mentioned.
We selected 623 images (out of over 10000) for the
labeling procedure. The labeling was done manually using
a simple MATLAB-tool. Doing so, 479 samples were
produced for the experiments. The distribution of samples
is the following: 23 Stratus (St), 71 Stratocumulus (Sc),
27 Cumulus (Cu), 29 Cumulonimbus (Cb), 9 Nimbostratus
(Ns), 42 Altostratus (As), 68 Altocumulus (Ac), 2 Cirrocumulus (Cc), 65 Cirrus (Ci), 33 Cirrostratus (Cs) and 109
sky samples.
A sample of cropped input image and result images (obtained by applying the results of the labeling process to original image) in one example case are shown in Fig. 4.
Because of the fact that one the authors had more than
adequate weather observation training and operational experience on traditional cloud classification, we were able to
carry out the labeling without outside assistance. The task
was time to time quite technical without the help of cloud
height information, but not so tedious as often described.
Possibility of misclassification still exists.
3.2. Tests and Results
Features were extracted globally throughout the input image
— the black areas, seen in sample image (see Fig. 4) presenting the extraction of samples, are discarded in the calculation. The current implementations of multi-scale LBPF
and multi-scale LBP, that are well equipped to capture largescale structures, were time consuming to calculate. Results
were promising and therefore worth to wait.
Two different classification set-ups were used. In the
first, all the ten cloud categories and the sky were tried to
be classified. In the second set-up, five categories (St, Cu,
Ac, Ci, and clear sky) similarly to [4] were considered to
perform some comparison.
The results obtained with different LBP and LEP operators are collected in Table 1. It shows the mean classification
rate for each texture operator in both classification tasks. In
addition it shows the number of bins used in each feature
histogram. The prefix MS in the table refers to multi-scale
approach.
Though it is agreed that edges contain a considerable
amount of information to distinguish different clouds automatically, LEP did not solve the problem quite the way
we hoped. There are, for instance, found edges from clear
sky and almost transparent high clouds are occasionally left
undetected by the edge detector. Even though improvement in resolution in the imagery might in itself help, features are not working as desired. The threshold value used
in the pre-processing of images before calculating LEP is
very important in LEP feature extraction. It will require
more research on what kind of pre-processing should be
used with LEP. Again, attached to the low resolution, the
fibrous structure of high clouds were occasionally out of
reach for LEP. Smooth clouds (like Altostratus and Cirrostratus) with edges in infinity were similarly problematic.
We have selected here representative classification results. Figs. 5 and 6 represent confusion matrices obtained
by extracting features with LBPu2
8,1 -operator. Figs. 7 and 8
illustrate classification results achieved by performing feature extraction with multi-scale LBPu2
8,1+16,2+24,3 . Numerical values contained by the confusion matrices are percentage values.
LBP is able to extract well discriminative features also
St Sc Cu Cb Ns As Ac Cc Ci Cs Sky
St Sc Cu Cb Ns As Ac Cc Ci Cs Sky
8
10
6
2
0
2
7
1
St
13
1
3
0
3
6
1
Sc 23 36 26 16
5
13 19
0
12 2
0
Sc 14 37 20 10 12
9
20
0
9
1
0
Cu
1 10 35 18
0
0
2
0
0
0
0
Cu
4 10 44 23
0
10
4
0
1
0
0
Cb
1
19 31
0
6
5
44
4
2
0
Cb
8
7
18 40
1
10
4
0
1
0
0
Ns 11 2
0
0
40
6
1
0
0
2
0
Ns
6
2
0
0
23
5
1
0
0
1
0
As 11 11
3
5
29 32
9
0
2 10
2
As 14 7
3
6
35 41
5
0
3 16
0
Ac
2 19
9
14 13
9
43
0
14 3
0
Ac
0 19
7
10 13
5
54
0
44 7
0
Cc
0
0
0
0
1
0
2
0
Cc
0
0
0
0
0
0
0
0
3
0
0
6
3
0
9
7
100 68 16
2
St
20 6
5
0
0
0
0
24 7
Ci 11 8
6
7
0
10 14 56 50 21
7
Ci 22 10
Cs 11 3
0
0
2
4
12
0
7 32
6
Cs
Sky 10 0
1
1
2
7
0
0
6 19 83
Fig. 5. The confusion matrix obtained with ten cloud genres
and the sky class using LBPu2
8,1 features.
St Cu Ac Ci Sky
St
31
9
10
9
1
Cu 10 64
6
1
0
Ac 19 17 60 20
1
Ci 22
9
22 61
9
Sky 17
1
3
89
9
Fig. 6. The confusion matrix obtained with the four cloud
genres and the sky class using LBPu2
8,1 features.
from thin and smooth clouds. Understandable confusion
can be observed in couple of situations. Certain forms of
Altocumulus and Stratocumulus clouds with similar kind
of appearance are misclassified in some cases — they are
both layered clouds, Stratocumulus being broader. Cloud
base height information would bring solution to this problem. The fact that Cirrocumulus is classified as Cirrus, as
it is in the confusion matrix in Fig. 7, is quite satisfactory. The Cirrocumulus is often quite a short term form.
Here the misclassification is tied to the fact that there were
only 2 samples of this cloud genre — zero samples for training and two for testing for example. Cumulonimbus (incus)
consists of veil-like top (formed of ice crystals) that resembles (and also acts as origin of thick) high clouds. There
are, of course, Cumulus like forms in Cumulonimbus. Cumulonimbus clouds evolve from Cumulus clouds, but have
more massive vertical dimension. These facts indicate that
LBP features have no significant problems. These are more
2
5
3
1
0
2
3
12
3
0
7 49
2
Sky 5
0
0
0
0
5
1
0
2
95
3
Fig. 7. The confusion matrix obtained with ten cloud genres
and the sky class using LBPu2
8,1+16,2+24,3 features.
St Cu Ac Ci Sky
St
36 12
9
8
1
Cu
8
70
5
3
0
Ac 21
8
70
7
0
Ci 32 10 15 78
2
Sky 3
0
1
4
97
Fig. 8. The confusion matrix obtained with the four cloud
genres and the sky class using LBPu2
8,1+16,2+24,3 features.
modelling related problems. LBP works quite well as a feature extractor in the cloud classification problem. The problems that arise from the number of samples per class appear
especially in the five class classification experiment.
Filtering removes some unwanted elements from images. Down side is that some useful elements are also faded
away due to the resolution. Usually, processing with multiple scales helps in incorporating more information about the
texture and makes the approach more robust to scale variations as well. The multi-scale LBP features indicated very
promising results in the cloud texture classification. This is
reasonable because clouds appear in very broad scales in the
images.
4. CONCLUSIONS
Two things, clouds and texture, are brought here together —
either of them have a definition with unanimous acceptance.
Texture is a salient elementary feature in natural scene
images. Here it manages to distinguish different cloud genres with very good performance by harnessing the characterization powers of local patterns.
The approach indicated a very good performance in the
classification experiment, where five classes (four cloud
genres and clear sky) were considered. The results were
promising also in the more difficult case where all the ten
cloud genres and clear sky were considered. It is, though,
safe to say that the cloud characterization problem can
not be solved entirely by using just textural features — no
matter how large amount of human labeled training data is
available.
Peura et al. [3] were able to classify all cloud genres
except Cumulonimbus with a good accuracy from all-sky
imagery reaching classification rate of 65% (correct classifications). The performance that they have achieved is mostly
due to their extensive and well-thought-out set of features.
As their imaging setup, sample set and generalization used
to present the results of the classification are so different
from our approach, it is difficult to make direct comparison.
The features that Peura et al. have developed could also be
applied in our research, but it was decided to limit to local
texture information only. The aim was to propose new features to supplement the existing ones. Our feature extractor
could be a worthy addition to those presented in [3].
Buch and Sun [4] reported a overall misclassification
rate of 39%. With that, we are able compete with our most
efficient methods in terms of classification rate.
Surprisingly, the uniform sky texture was not that easy
to distinguish as it would be expected. The use of adaptive
color or texture based segmentation might help to segment
the clouds automatically from the sky views. However, better quality images would be more than welcome in our approach anyway.
It would be reasonable to suggest that cloud images, for
a more comprehensive study, should be obtained simultaneously with weather observations, and classified manually
immediately after taking them. Cloud height information
(Ceilometer data) should also be collected at the same time
as well to make verification of the labels given later on.
It might be useful to create natural language descriptions
automatically from sky images, e.g. descriptions of regions
in an image. Each region could, for example, get the label
that best describes that area, e.g. which cloud class has the
majority over others in the section that is currently being
processed.
The fact that the classification results will remain quite
local (with this kind of arrangement) is not a disbenefit —
other variables like temperature and air pressure are also
measured locally throughout the globe. Though, a better
sky coverage using, for example, rotating camera would be
of interest in the future.
Acknowledgements The Finnish Meteorological Institute
is acknowledged for providing the sky images.
5. REFERENCES
[1] A. Visa, K. Valkealahti, J. Iivarinen, and O. Simula,
“Experiences from operational cloud classifier based
on self-organizing map,” in Application of Artificial
Neural Networks, SPIE 2243, 1994, pp. 484–495.
[2] A. Visa and J. Iivarinen, “Evolution and evaluation
of a trainable cloud classifier,” IEEE Transactions on
Geoscience and Remote Sensing, vol. 35, no. 5, pp.
1307–1315, September 1997.
[3] M. Peura, A. Visa, and P. Kostamo, “A new approach
to land-based cloud classification,” in International
Conference on Pattern Recognition, 1996, pp. 143–
147.
[4] K. Buch and C. Sun, “Cloud classification using
whole-sky imager data,” in 9th Symposium on Meteorological Observations and Instrumentation, 1995,
pp. 353–358.
[5] T. Ojala, M. Pietikäinen, and T. Mäenpää, “Multiresolution gray-scale and rotation invariant texture classification with local binary patterns,” IEEE Transactions
on Pattern Analysis and Machine Intelligence, vol. 24,
no. 7, pp. 971–987, 2002.
[6] T. Mäenpää and M. Pietikäinen, “Texture analysis
with local binary patterns,” in C. Chen and P. Wang
(eds.) Handbook of Pattern Recognition and Vision.
2005, pp. 197–216, World Scientific, 3rd Edition.
[7] C-J. Liao and S-Y. Chen, “Complementary retrieval
for distorted images,” Pattern Recognition, vol. 35,
no. 8, pp. 1705–1722, August 2002.
[8] M. Turtinen and M. Pietikäinen, “Contextual analysis
of textured scene images,” in British Machine Vision
Conference, 2006, pp. 849–858.
[9] T. Mäenpää and M. Pietikäinen, “Multi-scale binary
patterns for texture analysis,” in 13th Scandinavian
Conference on Image Analysis, 2003, pp. 885–892.
[10] C-H. Yao and S-Y. Chen, “Retrieval of translated, rotated and scaled color textures,” Pattern Recognition,
vol. 36, no. 4, pp. 913–929, April 2003.
[11] G. Finlayson, S. Chatterjee, and B. Funt, “Color angular indexing,” in European Conference on Computer
Vision (ECCV), 1996, pp. 16–27.