36 10 2015 10 Vol.36 No.10
October 2015
Journal on Communications
doi:10.11959/j.issn.1000-436x.2015263
1 1 2
(1. 3000712. 100191)
!"#$%&' UCI ()
*+(),-
. !"#/0&1.23.4
56789:
';<56()=.>?;<@ABCDE,-FG;,HIJKLMN(OPQ=.
RS&
T56T
TP181
A
Study of implicit information semi-supervised learning algorithm
LIU Guo-dong1, XU Jing1, ZHANG Guo-bing2
(1.College of Computer and Control Engineering, Nankai University, Tianjin 300071, China;
2. School of Electronic and Information Engineering, Beihang University, Beijing 100191, China)
Abstract: Implicit information semi supervised learning algorithm was studied. The implicit information semi supervised
learning algorithm was used in support vector machine and random forest, which were called semi-SVM and semi-RF.
The semi-SVM and semi-RF were evaluated by using UCI, the experimental results show that the semi-SVM and
semi-RF are more effective and more precise. The semi-SVM and semi-RF were applied to classifying lung sounds, and
verified the effect by using the actual lung sounds data. the quantity and quality of samples affect semi-SVM and
semi-RF were analyzed.
Key words: semi-supervised learning; lung sounds; implicit information
1
!"#$
%&'()
$%&'(*+,-./0123)$4%
&'(56789:.[1~3];<=
>?@A"BCDEFGHI;
(JKL656MN
OPQRS34%&'(T%&'(HU
VWXYZ[\4%&'(]^%&OP;_
`23%&`'(\Ma]^bcd9M
aMa:.;efg[4h6]"ijbc
(self-training)FKkldm`nOPo>FK[7]p
qrstFK [8,9] puvbc
K [10,11]wxyFKz{|dVWQ
R?}~\4%&'(]^%&;|
€aQ
R [12] pstQR[13]‚ƒI„:QR[15];
q%&KFK…†‡ˆ‰q Š‹2
3#$%&'(\)$%&'(]^%&Œ
Ž1‘Š%&’“”•e–—>'(56
˜™š—>›œbcMa:.ž;
fg[15]eŸI ¡¢q£¤¥¦§¨
© FSSL, formative semi-supervised
learningªFK;«FK¬M23T%&'(%
&'(HU£¤X­XY®xWX­XY¯?
T%&'(°±§¨²>³
%&'(
´;®µ±§¨`¶·³bc'(´B³]^
bc·i³Ma;q£¤¥¦¸:
FK¹º»¼QR\4%&'(]^
%&½¾¿ŒÀ¬MpÁÂ123º4%&'
(¥¦;
2015-05-14T2015-09-27
2015263-1
134
(f|DE FSSL FK;® FSSL FK
Ã3q}ÄG$©SVM, support vector machineª
ÅÆÇ©RF, rondom forestªOP;23»3
UCI 56È"56Éʺ FSSL FK9Ma
Ë¢ÌÍ;ŸIFÎÏ\ !ÐÑ56"+
)$%&'( !”•ÒÓ®q FSSL F
K SVM RF OPÃ3qÐÑÔÕ@A; ÉÖ
Í×Ø«FKŽd\%&ÐÑ56]^ÙÚ
Ma;
2
!
36 %&'(´"'(HU¢$XY
uj
θ = θl î? 2 X'(HUþ"
i
K
P ( li , u j ) = P ( li ) P ( u j | li ) = P ( li ) ∑ P ( u j | zk )P ( zk | li )
k =1
(2)
23
P ( zk | li , u j ) =
2.1
R S S 5 6 ' ( ´ S = ( X ,Y ) =
n
{( xi , yi ) xi ∈ R , yi ∈ R, i = 1, 2,, S } S = L + U ?
P ( li zk ) P ( zk )
P ( zk | li ) =
P ( li , u j zk ) P ( zk )
∑ P ( li , u j | zt )P ( zt )
âq L U d£¤¥¦¸: Z ?…
öé[ñd©4ªŽdä?
P ( zk | li , u j ) =
%&56 U = { x1 , x2 ,, x j ,, xU } Ü L U ;
\q–ÝXÞßàÁÂ1234
P ( li zk ) P ( u j | zk ) P ( zk )
∑ P ( li | zt ) P ( u j | zt )P ( zt )
L6©2ªh©5ªŽdú©1ª]Iä?
ρ = log P ( L,U , Z )
áDEßභ L U X­¸:;
K
âq L U znj¿IMN'(´<=Ž
L
U
{
}
= ∑∑∑ θli j log[ P ( li | zk ) P ( u j | zk ) P ( zk )] ⋅
dã¯zÅä$ Z åo;\qæI
u
k =1 i =1 j =1
çè Z k z
P ( li | zk ) P ( u j | zk ) HéX
(
)
K
L
U
(
u
k =1 i =1 j =1
<=ì{|í P( zk | li ) î_`®xy£¤¥¦
<?
¸:îïði L ´"'(¸:"³
%&
∑ P ( z ) = 1 ∑ P (l | z ) = 1 ∑ P (u
'(´"ê_뤺4%&'(¥¦ß× 1 ñò;
K
L
i
k =1
i =1
f2
fs
Y
P ( z1 | li )
P ( zk | li )
)î;
Λ ( ρ , λ , µ ,ν ) = ρ + λ ∑ (1 − P ( zk ) ) +
k =1
L
U
i =1
j =1
L
U
∑∑θ
P ( zk ) =
U
k =1 i =1 j =1
u
(
L
U
uj
li
i =1 j =1
U
)
P ( zk | li , u j )
∑∑θ
k =1 i =1 j =1
L
uj
li
i =1 j =1
ρ = log P ( L,U , Z ) = ∑∑∑ θ log P ( li , u j , zk )
K
(
\©7ªí?
uj
li
= ∑∑∑ θli j log[ P zk li , u j P ( li , u j )]
| zk ) = 1
µ ∑ (1 − P ( li | zk ) ) + ν ∑ 1 − P ( u j | zk )
Š,-óôõ?ßàí`Ʌö
P( zk | li ) î;L6›)÷_øù P( zk | li ) îŽ
dú P( L,U , Z ) îûi›)ü log P( L,U , Z ) ý›
U
j
j =1
K
f1
L
k
(6)
ñdŽd²ZI 5
li
K
U
k
L
)
P zk li , u j + ∑∑∑ θli j log[P zk li , u j ]
­<= P( zk | li ) "ê_뤺´ U "'(¥¦;
(5)
K
t =1
%&'(´ U = { x1 , x2 ,, x j ,, xU } ¥¦;ÎÒ
(4)
K
t =1
ۂMMÕ?T%&56 L = { x1 , x2 ,, xi ,, xL }
1
(3)
P ( li )
(1)
uj
li
þ"θ ( i = 1,, L, j = 1,,U )×ò?T%&'(´
2015263-2
∑θ
P ( li | zk ) =
uj
li
j =1
L U
∑∑θ
i =1 j =1
P ( zk | li , u j )
uj
li
P ( zk | li , u j )
)
(7)
10 L
∑θ
P ( u j | zk ) =
L
uj
li
i =1
U
∑∑θ
L6fg[15]ÉÊÖÍI/ K ?ñMa'(
´56Wa 2 01 ∇ x ý12î
P ( zk | li , u j )
uj
li
135
P ( zk | li , u j )
1;
i =1 j =1
2.2
3
\qï𣤥¦¸:î`'(bc·i
}ÄG$Ma? f ( L' ) 23 f ( L' ) K\
Ma'(]^Ma<?¿'U
ñd{|®Ma'(£¤¥¦¸:í
núñUI„;
\ M a ' ( ´ Ž d 3 Î K]
^í‹?º #!$3"÷ F
KíMa'(´£¤¥¦¸:îßr 1
ñò;
ᅲ偠ߚᵤ
(3 Éë4 2 ‚MÒá23 UCI 56
È"56ÉÊq£¤¸:FK
Ë¢_`®FKÃ3qÐÑ56
ÔÕÉÊþ !56Ã3ÌÍ;
3.1 UCI (3e UCI 56È"7 9 W56*3
89ÉÊFKMÕ\:FK]^ÉÊ;
SVMpSVM RF É56mÖÍ\
ß× 2 ñò;
2
1
SVM
SVM
RF
Haberman*
156
100
50
75.22
75.78
75.24
Liver*
175
100
70
70.75
72.14
70.56
Breast
479
120
100
95.83
96.12
95.69
Ttt
558
200
200
93.50
93.54
89.15
600
200
200
83.95
86.78
77.89
Spect heart
147
70
50
83.95
86.18
81.28
Chess
1 696
1 000
500
92.45
94.57
91.57
Sonar*
108
60
40
75.78
83.25
83.63
!"#$
German
redit*
\qMa'( x 23#$›" ∇ x T
%&'(C¯?%! P ( zk | x) &6L6
Ždí·
P ( x | zk ) P ( zk )
P ( zk | x ) =
K
∑ P(x | z )P(z )
t
t
t =1
∇x
∑ P (l
i
| zk ) P ( x | li ) P ( zk )
i ,li ∈∇ x
=
K
∇x
∇x
∑ ∑ P (l | z ) P ( x | l ) P ( z )
i
t
t
i
t
i ,li ∈∇ x
þ" P( x | zk ) ×òMa'( x ›" ∇ x T
%&'(C'(î)*ü
∇x
∑
i ,li ∈∇ x
SVM SVM RF P ( li | zk ) P ( x | li ) P ( zk )
∇x
P ( x | li ) × ò M a ' ( x T % & ' (
li ( li ∈ ∇ x ) HUX­¢îŽd23 2 '(
HU)*;
«K" K p∇ x 2 +5îÚè,MXÞ
ßÍýî-ó‘ž.ÌÍ;
e ÉM;·<=ÖÍ(
z>q}ÄG$<=ÖÍ;?î)2
@ºT%&'(4%&'(HUX­¢
)2ó%&'(A
B#³¥¦;C¼
ÉM;Ždãq£¤¥¦¸:
FK
Ì;
3.2 3.2.1 DEŌz]^FGHIJKE¿L
M]^FN^?xI¼6ЂOP­Y
IQé¯3åoRWSÑxySÑ"
¤
TU¥¦
VW
oWxy¥
¦\qDEЂVWÑI B|X%;Ð
Ñ¥YIWZ[:¥Y\
]^:pB':p
Å:w§C<=ßà´23ÐÑ56
\DE Ђ_.]^ `< <Ia\
b
2015263-3
136
c:5¯;
\qЂSÑMaÔÕDEBdnIe
fiD9¢Xgú3ÔÕFKhiŠ
\qÐÑÔÕMaKMa&6I§
ÕÅFD5j.k±
DElT®M
aÃ3qÐÑÔÕ@A;»¼\ÐÑMa
nÔՍ¿ÐÑIWB|ÐÑÔÕFK;
\qMaFK–ÝbcMaŒ|íbc
'(ë¤m¸:;ŽÐÑ56"‘+)$
%&56ün
%&oЂ+,56xy56"뤺
p{|
qrý;Ï\x ,-(fýq FSSL
FK SVM RF OP ŒHs)$%&'
(ñ¤
³¥¦9 SVM RF Ã3qÐÑ
ÔÕ@AÃ3:.;
= É"23ÐÑÔÕ,-ÉÊq£¤
¥¦FK !Ã3ÌÍ;ÐÑ56
?njqjR!kkÐÑ56´Yñ
´56GHtuv}ĝ\ñ´Dw
1Apxypdzz]^º{|}Ê56~
:p
Ì:)×:;´D51 826 €
2
36 ´‚€9 © ‚©FTª
p‚‚©RUª
pƒ
‚©LUªp‚"‚©RMªpƒ"‚©LMªp‚‚
pƒ‚©LDª
p‚`‚©RBª
pƒ`‚©LBª
ª
©RDª
´„56$ûi 49 302 '(56;
3.2.2 OÔÕ"§¨ý\qMaÌÍ0…
QFè:¯3;
er 2 r 3 "†òº‡ˆÙ‰Š‰Ð
‚‹Ðьr\ŒrŽdãى
Š‰Ð‚Ðь
Ž)?ŠŒ§¨
eøl¥Yeä–nøl¥¦}10<
=(fDE"3Œr"§¨¯?ÐÑ
ÔÕ§¨;
É"\ىŠ‰Ð‚ñ\ÃÐ
ћ)_öŒö©PFªp'(Œö©MNFªp
Ðф‘¢©LSIªp„_ö 25%p50%p75%
Œö Q25pQ50pQ75 ?Š]^ºM;;e
Ɏdãxy+5 2 aÐÑ56+
FIè?Šßr 4 ñò<=(f"7
ý PFpMNFpQ25pQ50pQ75 ¯?ÐÑÔÕ
§¨+5;
(RU_nonNormal)
2015263-4
!"
10 #
$%&'()*+,-./012
3
(RU_Normal)
3
4
3.2.3
RU 34"5678
1) ÔÕÌÍM;
= É"® SVM OPm RF
OPÃ3qÐÑÔÕ@A;= É"23 20%
'(¯?
%&'(’® 30%'(m¸
:“”¯?%&'(’30%'(¯?<='
(; ÉÖÍß× 3 ñòe ÉÖ͎dãâ
q³¥¦•ð SVM Ë¢%– SVM
Iè9;
2) ROC —[
L6 !Ã3{í\qÐÑÔÕ,-D
„˜™|š”Ђ
,-'(;<=
137
!"
SVM SVM SVM 9:/%
SVM 9:/%
RF 9:/%
FT
72.50
76.67
73.67
LB
72.18
74.44
74.49
LD
65.56
66.73
65.23
LM
69.58
71.91
70.67
LU
60.42
61.46
61.05
RB
63.33
69.44
67.31
RD
76.67
78.92
79.56
RM
72.22
75.00
75.04
RU
87.96
88.77
89.65
É"RÙ‰ÐÑ?ÙaىÐÑ?›a;
TP ×òÙÚÔՍىÐÑ'(.œü"
ö’FP ×ò’MىÐÑ'(”•ü“žö;
Î»¼ ROC —[M;ÐÑÔՌ"ö
“žöHUXYþ"
2015263-5
FP =
TP =
138
e× 4 Ždãq FSSL Žd9"öüÔÕZىÐÑ'(.
œ‹\“žön
Ÿä;
4
RU
TP/%
FP/%
SVM
91.75
12.78
SEMI_SVM
93.44
12.78
SEMI_RF
92.19
12.78
SVM ;</%
SVM 7=9:/%
5
87.96
10
87.96
15
87.99
20
88.03
25
88.25
30
88.77
36 ú'(Cª«ŸLIae–•ðº’MC
ú SVM :.ž;â=Ž¬ SVM
MaË¢%&'(¦$XYOP\ !Ã
3\
Iè ¡;
4
3) ¡<¢M;
Î23Ð56M;%&'(5
$ SVM :.XY ÉÖÍß× 5
ñò;e ÉÖ͎dãì
%&'($
•ðiIè6¢Œ£\ SVM :.
¡;
5
㒧ᴳ䇁
(fDEºq£¤¥¦¸:
FK*®þÃ3q SVM RF OP;?º
ÉÊq£¤¥¦ SVM RF
KÒá23 UCI 56È\KË¢]^º
ÉÊ_`23 !´ÐÑ56ÔÕ,Éʺþ !56,-ÌÍ;¿ŒM;º
4%& '(5$dm4 %& '(¦$ q£¤ ¥¦ FK XY;Žd ã
%&'(5$¦$\q=FK !
Ã3ÌÍ ¡1)%&'(5$\qMa:
. ¡þ¦$ ¡|);<=}Ê\
Iè5$%&'(”•=FK\q
!Ã31./ !Ã3"k­þ
§>®;
খ㗗᭛⤂˖
[1]
YANG L X, YANG S Y, Semi-supervised hyperspectral image
classification using spatio-spectral laplacian support vector machine[J]. IEEE Geoscience and Remote Sensing Letters, 2014,
11(3): 651-656.
[2]
SVM :.*Iè9q SVMþ
:.¤ ý q% &'( o . ¥ ³¥ ¦
$;?ºÉÊx :¦øl SVM ñ·}
ÄG$#$o>IyÅ'(¯?%&
ÐÐÑ'(23xy'(]^
ɧ„ÖÍß× 6 ñò; ÉÖÍ×Ø}ÄG
$ ¨ " C ñ. ¥ ³¥ ¦ ¨ # \ SVM :.‡©n
¡;‹ÅF•ð
cation with universum learning[J]. IEEE Transactions on Cybernetics,
DOI: 10.1109/TCYB.2015.2403573, 2015.
[3]
IEEE Transactions on Medical Imaging, DOI: 10.1109/ TMI.
2015.2404572, 2015.
[4]
SVM 7=9:
0.01
87.96
ZHOU Z H. Co-training paradigm in semi-supervised learning[A]
Proceeding of the Chinese Workshop on Machine Learning and Applications[C]. Nanjing, ChinaE2007. 261-267.
SVM >?@ABCD<@A
ONOFREY J A, Low-dimensional non-rigid image registration using
statistical deformation models from semi-supervised training data[J].
[5]
6
LIU L C, HSAIO W H, LEE C H, et al. Semi-supervised text classifi-
ZHOU D YEBOUSQUET OELAL T NEet al. Learning with local and
global consistency[J]. Advances in Neural Information Processing
System, 2004E3(21):23-28.
[6]
WU C MEWANG X D, BAI D Y. Fast incremental learning algorithm
of SVM on KKT conditions[A]. 2009 Sixth International Conference
0.05
88
0.10
88.23
0.50
87.46
on Fuzzy Systems and Knowledge Discovery[C]. 2009. 551-554.
[7]
JAKKOLA T, HAUSSLER D. Exploiting generative models in discriminative classifiers [A]. Advances in Neural Information Processing
Systems[C]. Cambridge, MA: The MIT Press, 1999. 487-493.
2015263-6
10 #
[8]
$%&'()*+,-./012
!"#$
FRALICK S C. Learning to recognize patterns without a teacher[J].
IEEE Transactions on Information Theory, 1967,13(1):57-64.
[9]
139
*&+1966-5678
#$%&")019
:;<
AGRAWALA A K. Learning with a probabilistic teacher[J]. IEEE
Transactions on Information Theory, 1970, 16(4):373-379.
[10] HOLUB AEWELLING M, PERONA P. Exploiting unlabeled data for
hybrid object classification[A]. Proc of the 17th Annual Conference on
Neural Information Processing Systems[C]. 2005.165-171.
[11] WANG WEZHOU Z H. Analyzing co-training style algorithms[A].
Proc of the 18th Conference on Machine Learning[C]. 2007.454-465.
[12] KUBAT M, HOLTE B C, MATWIN S. Machine learning for the detection of oil spills in satellite radar images[J]FMachine LearningE
1998,30(2):195-215.
()1967- !
"
#$%&'(")*+
%,-./0123014
[13] JOACHIMS T. Transductive inference for text classification using support
vect ormachines[A]. Machine Learning-international Workshop Then Conference[C]. Morgan Kaufmann Publishers, INC, 1999. 200-209.
[14] BLUM A, LAFFERTY J, RWEBANGIRA M R, et al. Semi- supervised learning using randomized mincuts[A]. Proceedings of the
Twenty-first International Conference on Machine Learning[C]. ACM,
2004. 13.
[15] ZHU X Q. Cross-domain semi-supervised learning using feature formulation[J].IEEE Transactions on Systems MAN, and CybernerticsPart B: Cybernetics, 2011, 41(6):1627-1638.
2015263-7
%&'1979-