International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016), pp. 31-44
http://dx.doi.org/10.14257/ijseia.2016.10.2.04
Visible and Infrared Data Fusion
Gwanggil Jeon
Department of Embedded Systems Engineering, Incheon National University
119 Academy-ro, Yeonsu-gu, Incheon 406-772, Korea
[email protected]
Abstract
This paper shows data fusion method with visible light image and infrared image. We
assume an image is captured by FLIR image, which provides two images: visible light
and infrared images. The details of the image are obtained in visible light image. This
information is added to original low resolution infrared image and we assume this image
as the detail strengthen infrared image. Experimental results section provides PSNR and
MSE results. It is obvious that that proposed method with p=0.03 gives visually
satisfactory results.
Keywords: color image, membership function, quality enhancement, fuzzy model
1. Introduction
An optical image is captured by sensor which is transformed into a digital signal [1].
Thus, the camera is an optical instrument for remembering data, which could be saved in
a particular place, transmitted and transformed to another place [2, 3]. Recent
photographic camera developed from the camera obscura [4]. The functioning of the
camera is very close to the operating of the human eye. There are two cameras: visible
light camera and infrared camera [5]. The visible light camera is ordinary camera, while
infrared camera captures signal acts like thermometer [6]. The infrared radiation was
found in early 19th century and learned a type of invisible radiation in the spectrum lower
in energy than red signal, via its accomplish upon a thermometer.
For visible camera, it can work with the light of the visible spectrum or with other
locations of the electromagnetic spectrum [7, 8]. On the other hand, infrared camera
captures invisible radiant energy, electromagnetic radiation with longer wavelengths than
that of visible light. In general, most of the thermal radiation sent out by objects near
room temperature is infrared [9-11].
Generally speaking, infrared image has lower resolution while visible image has
higher resolution. Although infrared image can be enlarged by super resolution technique,
most important details are still missing. To achieve better performance, in this paper, we
use higher resolution visible image. We obtain details from visible image, and then this
information is added to upsampled infrared image. We first transform color image into
YCbCr. Details are obtained from luminance (Y) information.
This paper is arranged as follows. Section 2 describes the proposed method. Section 3
describes simulation results where objective and subjective performance are provided.
Conclusion remarks are provided in Section 4.
2. Proposed Method
Figure 1 shows an example of infrared images with different colormap. Four options
were used gray, hot, cold, and copper. The FLIR infrared camera has two outputs: one is
visible light image and the other is infrared image. Normally, visible light image has
ISSN: 1738-9984 IJSEIA
Copyright ⓒ 2016 SERSC
International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016)
higher resolution while infrared image has lower resolution. Therefore, object details can
be found from visible light image and which can be added to infrared image.
(a)
(b)
(c)
(d)
Figure 1. Examples of Infrared Images with Various Colormap: (a) Gray, (b)
Hot, (c) Cold, and (d) Copper
Figure 2 shows the flowchart of the proposed method. First of all, object is captured by
FLIR camera, which provides two images, infrared and visible light images. As visible
light image has higher resolution, detail of the images is obtained by using high pass filter.
On the other hand, the infrared image has lower resolution, it is required to upsample the
image to meet the size of visible light image. After upsampling it with super resolution
method, details from the high pass filter are added to provide visually satisfactory image.
There are few methods that we consider for super resolution method: nearest neighbor
(hNN), bilinear (hBI), and bicubic (hBC) as shown in Eqs. (1-3). Note that, we assume
upsampling rate is 2.
32
Copyright ⓒ 2016 SERSC
International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016)
Object
High resolution
Low resolution
Visible camera
Infrared camera
High pass filter
Detail achievement
with factor of p
Image upsampling to meet
visible camera image size
Image fusion
p
High resolution
infrared image
Figure 2. Flowchart of the Proposed Method
1
hNN  1
0
1
2

1
hBI  
hBC
1 0
1 0  ,
0 0 
2 1
4 2 
2 1 
(1)
(2)
,
4
1 4 6 4 1 
 4 16 24 16 4 


 6 24 36 24 6 


 4 16 24 16 4 
1 4 6 4 1 
.

64
(3)
To apply Eqs. (1-3), we first create zero-interleaved version of the matrix. Then the
matrix becomes double the size of the original and will include mainly zeros. We now
substitute the zeros by applying given spatial filters to this matrix.
Copyright ⓒ 2016 SERSC
33
International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016)
(a)
(b)
(c)
(d)
(e)
(f)
Figure 3. Edge Results to be Added in Low Resolution Image: (a) p=0.03, (b)
p=0.05, (c) p=0.07, (d) p=0.10, (e) p=0.20, and (f) p=0.30.
To achieve high pass filter results, we generated new 3-by-3 matrix hHPF, which is
obtained by Eq. (4).
hHPF
a
 p   d
 g
b
e
h
c
f  .
i 
(3)
where a+b+c+d+f+g+h+i=-e, and a, b, c, d, f, g, h, i are negative numbers and e is
positive number. We assume b, d, f, and h have more effect than a, c, g, i. Parameter p is
high pass filter strength parameter, and in this paper we assume it p=0, p=0.03, p=0.05,
p=0.07, p=0.10, p=0.20, and p=0.30, as shown in Figure 3.
34
Copyright ⓒ 2016 SERSC
International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016)
It can be found from Fig. 4 that bigger p results strong edges, small p provides weak
edges. By adding Figure 3(b) to original infrared image Figure 4(b), we obtain edge
strengthened image, Figure 4(c).
(a)
(b)
(c)
Figure 4. (a) Original Image, (b) Zoomed Result of Infrared Image, (c)
Proposed Method Applied Image (p=0.05)
The output image ImIRout is calculated as Eq. (X). ImV represents original visible light
image and hHPF*ImV shows convoluted results with hHPF.
Imout
IR  Im IR  hHPF *ImV .
(4)
3. Experimental Results
The proposed method was simulated under MATLAB 7.0 environment. Test images
are 25 Zahra dataset, which has 768-by-512 size. Infrared image has one channel, while
color image has 3 channels. Visual performance was compared and presented.
First of all 768-by-512 size infrared image is downsampled and upsampled using
bicubic method. Decimation factor was 2, therefore downsampled image size was 384by-256. High pass filter parameters were a=-2, b=-3, c=-2, d=-3, f=-3, g=-2, h=-3, i=-2,
and therefore e=20.
Copyright ⓒ 2016 SERSC
35
International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016)
(a)
(b)
(c)
Figure 5. Three Original Images in Zahra Set: (a) #18, (b) #19, and (c) #20
Figure 5 shows three original test images: #18, #19, and #20, which are infrared
images. The subjective performance comparisons on Figure 5 are shown in Figures 6-8.
Each figure has six images: low resolution image (p=0), edge enhanced images with
parameter values p=0.03, p=0.05, p=0.07, p=0.10, and p=0.20. It can be found in all (a)
images that details in these images were missing. As p increases, details of images are
getting distinct. However, big p does not always give the best performance. As can be
seen in (f) image, some pixels were over-sharpened.
(a)
36
(b)
Copyright ⓒ 2016 SERSC
International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016)
(c)
(d)
(e)
(f)
Figure 6. Subjective Performance Comparison on #18 Image: (a) Low
Resolution Image, (b) p=0.03, (c) p=0.05, (d) p=0.07, (e) p=0.10, and (f)
p=0.20
(a)
Copyright ⓒ 2016 SERSC
(b)
37
International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016)
(c)
(d)
(e)
(f)
Figure 7. Subjective Performance Comparison on #19 image: (a) Low
Resolution Ismage, (b) p=0.03, (c) p=0.05, (d) p=0.07, (e) p=0.10, and (f)
p=0.20
(a)
38
(b)
Copyright ⓒ 2016 SERSC
International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016)
(c)
(d)
(e)
(f)
Figure 8. Subjective Performance Comparison on #24 image: (a) Low
Resolution Image, (b) p=0.03, (c) p=0.05, (d) p=0.07, (e) p=0.10, and (f)
p=0.20
Two objective performances metrics were used in this paper. One is PSNR and the
other is SME. Table 1 shows PSNR results for 25 Zahra dataset. The last row shows
average results of each parameter p. It can be found from Tables 1 and 2 that p=0.03
provides the best performance. The second best p was p=0.05.
Table 1. PSNR Results (in dB) for 25 Zahra Images
1
2
3
4
5
6
7
8
9
10
11
12
p=0
p=0.03
p=0.05
p=0.07
p=0.10
p=0.20
p=0.30
32.929
35.524
33.412
30.487
26.896
19.938
16.053
34.666
35.762
35.516
34.568
32.586
26.807
23.058
29.806
32.042
32.588
31.922
29.629
22.787
18.674
32.026
34.139
35.245
35.656
34.646
28.098
23.720
27.980
29.134
29.260
28.788
27.339
21.963
18.197
32.539
33.195
33.279
33.053
32.238
28.250
24.876
28.151
30.809
32.188
32.429
30.532
22.953
18.515
34.524
36.281
35.840
34.335
31.639
25.105
21.216
31.081
32.129
32.228
31.785
30.430
25.253
21.541
30.135
32.967
33.392
32.071
29.001
21.677
17.549
29.102
30.556
30.984
30.781
29.513
23.955
20.025
34.060
35.323
35.646
35.416
34.236
28.995
25.168
Copyright ⓒ 2016 SERSC
39
International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016)
13
14
15
16
17
18
19
20
21
22
23
24
25
Avg.
27.066
28.867
28.357
26.765
23.996
17.417
13.527
31.566
32.278
31.017
29.184
26.509
20.378
16.676
32.366
36.854
39.725
38.991
34.208
24.997
20.489
30.785
34.416
36.969
37.858
34.689
25.325
20.645
34.763
34.897
33.754
32.201
29.870
24.187
20.613
33.849
35.477
35.962
35.716
34.270
28.373
24.374
31.560
34.016
34.738
34.120
31.709
24.615
20.438
38.297
39.116
38.434
37.111
34.843
28.983
25.293
39.813
42.589
41.703
39.344
35.893
28.803
24.836
45.281
39.370
35.614
32.885
29.883
23.918
20.401
37.565
39.680
38.878
36.887
33.788
26.995
23.085
27.119
27.046
26.130
24.874
22.888
17.650
14.197
36.571
38.236
37.632
36.016
33.272
26.767
22.904
32.944
34.428
34.340
33.330
30.980
24.567
20.643
Tables 1 and 2 are re-drawn in figures as shown in Fig. 9. It is noted that results of
p=0.20 and p=0.30 are excluded as their MSE value is too large to show.
(a)
40
Copyright ⓒ 2016 SERSC
International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016)
(b)
Figure 9. Objective Performance Comparison for Zahra set: (a) PSNR (in
dB), (b) MSE. It is noted that results of p=0.20 and p=0.30 are not Included
as their MSE Value is Very Large.
Table 2. MSE Results for 25 Zahra Images
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
p=0
p=0.03
p=0.05
p=0.07
p=0.10
p=0.20
p=0.30
33.126
18.225
29.639
58.133
132.897
659.642
1613.361
22.206
17.255
18.261
22.713
35.851
135.633
321.553
67.994
40.637
35.831
41.771
70.830
342.313
882.443
40.787
25.069
19.435
17.678
22.311
100.747
276.095
103.545
79.366
77.112
85.950
120.005
413.773
984.848
36.241
31.157
30.561
32.198
38.842
97.291
211.588
99.545
53.975
39.292
37.168
57.527
329.456
915.333
22.944
15.311
16.948
23.966
44.581
200.734
491.402
50.698
39.829
38.930
43.107
58.892
194.007
456.044
63.037
32.841
29.776
40.363
81.840
441.945
1143.351
79.964
57.208
51.842
54.318
72.738
261.584
646.501
25.530
19.090
17.719
18.685
24.519
81.953
197.832
127.772
84.416
94.921
136.951
259.107
1178.595
2886.234
45.341
38.481
51.451
78.457
145.281
596.101
1397.798
37.714
13.418
6.928
8.204
24.678
205.784
581.034
54.278
23.524
13.067
10.647
22.087
190.819
560.475
21.714
21.054
27.398
39.169
67.002
247.968
564.613
26.801
18.424
16.476
17.436
24.329
94.568
237.518
45.400
25.793
21.840
25.181
43.870
224.701
587.892
9.624
7.970
9.325
12.648
21.320
82.187
192.225
Copyright ⓒ 2016 SERSC
41
International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016)
21
22
23
24
25
Avg.
6.789
3.582
4.393
7.563
16.741
85.664
213.559
1.928
7.518
17.851
33.468
66.802
263.785
592.878
11.392
7.000
8.419
13.315
27.180
129.906
319.568
126.241
128.384
158.534
211.660
334.433
1117.049
2474.090
14.322
9.760
11.218
16.275
30.611
136.889
333.156
46.997
32.771
33.887
43.481
73.771
312.524
763.256
4. Conclusions
In this paper, we studied image data fusion method. We consider FLIR camera which
provides visible light and infrared images. We assume image edges (or details) can be
obtained from visible light image. The obtained edges are finally added to original low
resolution infrared image. Simulation results show that parameter p=0.03 gives the best
performance.
Acknowledgment
This research was supported by Basic Science Research Program through the National
Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future
Planning(2013R1A1A1010797). This paper is a revised and expanded version of a paper
entitled “Data Fusion for Infrared Image” presented at MulGraB2015, November 25-28,
2015 at International Center, Jeju National University, Jeju Island, Korea.
References
[1] H. Li, B.S. Manjunath and S.K. Mitra, “Multi-sensor image fusion using the wavelet transform,” IEEE
ICIP1994, vol. 1, (1994), pp. 51-55.
[2]
. Proen a, “On the feasi ilit on the visi le wavelength, at-a-distance and on-the-move iris
recognition,” IEEE IEEE S mposium Series on Computational Intelligence in Biometrics: Theor ,
Algorithms, and Applications, (2009).
[3] J. G. Daugman, “New methods in iris recognition,” IEEE Transactions on S stems, Man, and
Cybernetics - Part B: Cybernetics, vol. 37, no. 5, (2007), pp. 1167-1175.
[4] K. Park and J. Kim, “A real-time focusing algorithm for iris recognition camera”, IEEE Transactions on
Systems, Man and Cybernetics, vol. 35, no. 3, (2005) August, pp. 441-444.
[5] L. Ma, T. Tan, Y. Wang and D. Zhang, “Local intensit variations analysis for iris recognition”, Pattern
recognition, vol. 37, no. 6, (2004), pp. 1287-1298.
[6] Y. He, J. Cui, T. Tan and Y. Wang, “Ke techniques and methods for imaging iris in focus”, in
Proceedings of the IEEE International Conference on Pattern Recognition, pp. 557-561, (2006) August.
[7] J. R. Matley, D. Ackerman, J. Bergen and M. Tinker, “Iris recognition in less constrained
environments,” Springer Advances in Biometrics: Sensors, Algorithms and S stems, (2007) October, pp.
107-131.
[8] J. G. Daugman, “ ow iris recognition works”, IEEE Transactions on Circuits and Systems for Video
Technology, vol. 14, no. 1, (2004) January, pp. 21-30.
[9] J. L. Cam ier, “Iridian large database performance”, Iridian Technologies, Tech. Rep.,
http://iridiantech.com, (2007).
[10] G. Jeon, “Contrast Intensification in NTSC YIQ,” IJCA vol. 6, no. 4, (2013) August, pp. 157-166.
[11] G. Jeon, “Measuring and Comparison of Edge Detectors in Color Spaces,” IJCA vol. 6, no. 5, (2013)
October, pp. 21-30.
Author
Gwanggil Jeon received the BS, MS, and PhD (summa cum laude) degrees in
Department of Electronics and Computer Engineering from Hanyang University, Seoul,
Korea, in 2003, 2005, and 2008, respectively.
From 2008 to 2009, he was with the Department of Electronics and Computer
Engineering, Hanyang University, from 2009 to 2011, he was with the School of
42
Copyright ⓒ 2016 SERSC
International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016)
Information Technology and Engineering (SITE), University of Ottawa, as a postdoctoral
fellow, and from 2011 to 2012, he was with the Graduate School of Science &
Technology, Niigata University, as an assistant professor. He is currently an assistant
professor with the Department of Embedded Systems Engineering, Incheon National
University, Incheon, Korea. His research interests fall under the umbrella of image
processing, particularly image compression, motion estimation, demosaicking, and image
enhancement as well as computational intelligence such as fuzzy and rough sets theories.
He was the recipient of the IEEE Chester Sall Award in 2007 and the 2008 ETRI
Journal Paper Award.
Copyright ⓒ 2016 SERSC
43
International Journal of Software Engineering and Its Applications
Vol. 10, No. 2 (2016)
44
Copyright ⓒ 2016 SERSC