US 20050136549A1
(19) United States
(12) Patent Application Publication (10) Pub. No.: US 2005/0136549 A1
Gholap et al.
(43) Pub. Date:
Jun. 23, 2005
Related US. Application Data
(54) METHOD AND SYSTEM FOR
AUTOMATICALLY DETERMINING
DIAGNOSTIC SALIENCY OF DIGITAL
IMAGES
(60)
Provisional application No. 60/515,582, ?led on Oct.
30, 2003. Provisional application No. 60/530,174,
?led on Dec. 15, 2003.
(75) Inventors: Abhijeet S. Gholap, San Jose, CA
(US); Gauri A. Gholap, San Jose, CA
(US); C. V. K. Rao, San Jose, CA
(US); Sanford H. Barsky, Los Angeles,
CA (US); Prithviraj J adhav, San Jose,
CA (US); J ayant Abhyankar, San Jose,
CA (US); Madhura Vipra, Prune (IN)
Correspondence Address:
Publication Classi?cation
51
(52)
(57)
.
Im. C17
........................ ..
C12Q 1/00 ;
G01N 33/ 5 66
U.S. c1. ............................................................ ..436/501
ABSTRACT
A method and system for automatically determining diag
nostic saliency of digital images for medical and/or patho
LESAVICH HIGH-TECH LAW GROUP, P.C.
logical purposes. Luminance parameters (eg intensity, etc.)
SUITE 325
39 S. LASALLE STREET
to Which a chemical compound (e.g., a marker dye) has been
applied are automatically analyzed and automatically cor
rected if necessary. Morphological parameters (e.g., cell
membrane, cell nucleus, mitotic cells, etc.) from individual
components Within the biological sample are automatically
analyzed on the digital image. A medical conclusion (e.g., a
medical diagnosis or prognosis) is automatically determined
from the analyzed luminance and morphological parameters.
CHICAGO, IL 60603 (US)
(73) Assignee: Bioimagene, Inc., San Jose, CA
(21) Appl, No;
10/966,071
(22) Filed:
Oct. 15, 2004
from a digital image of a biological sample (e.g., tissue cells)
20
AUTOMATICALLY ANALYZE LUMINANCE PARAMETERS
FROM A DIGITAL IMAGE OF A BIOLOGICAL SAMPLE TO
WHICH A CHEMICAL COMPOUND HAS BEEN APPLIED
"\
22
|
AUTOMATICALLY ANALYZE MORPHOLOGICAL
PARAMETERS FROM INDIVIDUAL BIOLOGICAL
COMPONENTS WITHIN THE BIOLOGICAL SAMPLE ON THE ’\
DIGITAL IMAGE
24
AUTOMATICALLY DETERMINE A MEDICAL CONCLUSION
FROM THE ANALYZED LUMINANCE AND MORPHOLOGICAL —\
PARAMETERS
26
Patent Application Publication Jun. 23, 2005 Sheet 1 0f 29
US 2005/0136549 A1
FIG. 1
10
/
14
WINDOW 1
WINDOW 2
Patent Application Publication Jun. 23, 2005 Sheet 2 0f 29
US 2005/0136549 A1
FIG. 2
2°
@
/
AUTOMATICALLY ANALYZE LUMINANCE PARAMETERS
FROM A DIGITAL IMAGE OF A BIOLOGICAL SAMPLE TO
WHICH A CHEMICAL COMPOUND HAS BEEN APPLIED
"\
22
|
AUTOMATICALLY ANALYZE MORPHOLOGICAL
PARAMETERS FROM INDIVIDUAL BIOLOGICAL
COMPONENTS WITHIN THE BIOLOGICAL SAMPLE ON THE ’\
DIGITAL IMAGE
24
I
AUTOMATICALLY DETERMINE A MEDICAL CONCLUSION
FROM THE ANALYZED LUMINANCE AND MORPHOLOGICAL —\
PARAMETERS
26
Patent Application Publication Jun. 23, 2005 Sheet 3 0f 29
US 2005/0136549 A1
FIG. 3
28
/
AUTOMATICALLY ANALYZE LUMINANCE PARAMETERS
FROM A DIGITAL IMAGE OF A BIOLOGICAL TISSUE SAMPLE
TO WHICH A MARKER DYE HAS BEEN APPLIED TO
/'\
DETERMINE ONE OR MORE AREAS OF INTEREST IN THE
30
BIOLOGICAL TISSUE SAMPLE WITHIN THE DIGITAL IMAGE
I
AUTOMATICALLY ADJUST LUMINANCE PARAMETERS
WITHIN THE ONE OR MORE DETERMINED AREAS OF
A
INTEREST WITHIN THE DIGITAL IMAGE TO CREATE ONE OR 32
MORE ADJUSTED AREAS OF INTEREST
I
AUTOMATICALLY ANALYZE MORPHOLOGICAL
PARAMETERS FROM INDIVIDUAL BIOLOGICAL
COMPONENTS WITHIN THE ONE OR MORE ADJUSTED
AREAS OF INTEREST WITHIN THE DIGITAL IMAGE
\
34
I
AUTOMATICALLY FORMULATE A MEDICAL DIAGNOSIS
USING THE ANALYZED MORPHOLOGICAL PARAMETERS
WITHIN THE ONE OR MORE ADJUSTED AREAS OF
INTEREST WITHIN THE DIGITAL IMAGE
\
36
Patent Application Publication Jun. 23, 2005 Sheet 4 0f 29
US 2005/0136549 A1
38
FIG. 4A
46
‘:5
a,
E
q,
m
a,
l
i:
U’)
':
FIG 5A
FIG 5B
0
‘5
.
5
a5
_
Histlgram cf Original image
C
"5'59?" d "115581 "THE
3,
N
HID
In
.§.2
F"EH11
m.‘
1.1:: m
In
t
m
D
in
Il‘lIl]
O
N
@
511]
E
u
1 2! l3 5‘ H51D512714B1B19D21123253
m? '1' "Em
'
E’
1 B ‘5 51 m1111331551111992212t3
;
Gm; male lnhrlllt;
c
O
U
Patent Application Publication Jun. 23, 2005 Sheet 5 0f 29
US 2005/0136549 A1
FIG. 6
52
Histogram of Modified image
351101]
301100
Pix 251100
CO
20000
—— Series1
— Series2
“" 15000
10000
5008
v
1
23 45
67
8911113315517? 199 221 243
Gray scale intensity
Patent Application Publication Jun. 23, 2005 Sheet 6 0f 29
FIG. 9
US 2005/0136549 A1
68
Patent Application Publication Jun. 23, 2005 Sheet 7 0f 29
FIG. 10
US 2005/0136549 A1
Patent Application Publication Jun. 23, 2005 Sheet 8 0f 29
US 2005/0136549 A1
88
FIG. 12A
/
FIG. 12B
FIG. 12B
90
Patent Application Publication Jun. 23, 2005 Sheet 9 0f 29
US 2005/0136549 A1
94
AUTOMATICALLY SEGMENT MORPHOLOGICAL
COMPONENTS FROM A BIOLOGICAL TISSUE SAMPLE
WITHIN A DIGITAL IMAGE INTO ONE OR MORE AREAS OF
INTEREST
/\
96
AUTOMATIcALLY ADJUST VIEWABLE CHARACTERISTICS
OF THE sEGMENTED MORPHOLOGICAL COMPONENTS ’\
USING ONE OR MORE DIGITAL IMAGE PROCESSING
98
TECHNIQUES TO CREATE ONE OR MORE ADJUSTED AREAs
OF INTEREST wITHIN THE DIGITAL IMAGE
I
AUTOMATICALLY COMPUTE A MEDICAL DIAGNOSIS GRADE
BASED ON THE sEGMENTED MORPHOLOGICAL
COMPONENTS WITHIN THE ONE OR MORE ADJUSTED
AREAS OF INTEREST WITHIN THE DIGITAL IMAGE
10°
Patent Application Publication Jun. 23, 2005 Sheet 10 0f 29
US 2005/0136549 A1
FIG. 14A
102
AUTOMATICALLY COMPUTE PLURAL AREAS OF INTEREST
IN A BIOLOGICAL TISSUE SAMPLE WITHIN A DIGITAL IMAGE d
+
104
AUTOMATICALLY COMPUTE IMAGE STATISTICS FOR
THE DIGITAL IMAGE
706
I
AUTOMATICALLY ENHANCE A CONTRAST 0F COLOR
PLANES OF THE DIGITAI. IMAGE usING THE COMPUTED
IMAGE sTATIsTICs
A
V
108
I
AUTOMATICALLY COMPUTE A HISTOGRAM OF GRAYSCALE
VALUES FROM THE CONTRAST ENHANCED DIGITAL IMAGE A
110
I
AUTOMATICALLY DETERMINE PLURAL GRAYSCALE
INTENSITY VALUES AT A FIRST PEAK IN THE HISTOGRAM A
I
AUTOMATICALLY SEGMENT THE ENHANCED DIGITAL
IMAGE BY THRESHOLDING THE DETERMINED GRAY SCALE /\
INTENSITY VALUES WITH GRAYSCALE VALUES FROM THE
114
HISTOGRAM
I
AUTOMATICALLY COMPUTE COLOR CORRECTION VALUES
FROM THE BACKGROUND OF THE CONTRAST ENHANCED Q16
DIGITAL IMAGE
OA
FIG.
14B
Patent Application Publication Jun. 23, 2005 Sheet 11 0f 29
US 2005/0136549 A1
FIG. 14B
AUTOMATICALLY COMPUTE A PERCENTAGE OF PIXELS
INCLUDING ONE OR MORE PRE-DETERMINED CELLs 0F
/1\18
INTEREST IN THE coNTRAsT ENHANCED DIGITAL IMAGE
120
PIXEL PERCENTAGE
< FIRST VALUE?
YES
v
f
122
AUTOMATICALLY CLASSIFY BIOLOGICAL TISSUE SAMPLE
AS FIRST GRADE
124
PIXEL PERCENTAGE
< SECOND VALUE?
YES
I
f
126
AUTOMATICALLY CLASSIFY BIOLOGICAL TISSUE SAMPLE
AS SECOND GRADE
@
OB
FIG.
14C
Patent Application Publication Jun. 23, 2005 Sheet 12 0f 29
US 2005/0136549 A1
FIG. 14C
AUTOMATICALLY DETERMINE EDGES OF OBJECTS OF
INTERESTS IN THE CONTRAST ENHANCED DIGITAL IMAGE
I
AUTOMATICALLY DETERMINE EDGES OF COMPLETE
OBJECTS OF INTERESTS IN THE coNTRAsT ENHANCED
'
A
DIGITAL IMAGE
13°
i
AUTOMATICALLY coMPuTE A RATIO OF AREA OF
COMPLETE OBJECTS OF INTEREST To BACKGROUND
AREA
/\
132
134
COMPUTED RATIO <
PREDETERMINED
RATIO?
YES
v
/
136
AUTOMATICALLY CLASSIFY BIOLOGICAL TISSUE SAMPLE
AS A THIRD GRADE
r138
AUTOMATICALLY CLASSIFY BIOLOGICAL TISSUE SAMPLE
_
@3
AS A FOURTH GRADE
.
'
Patent Application Publication Jun. 23, 2005 Sheet 13 0f 29
FIG. 15
US 2005/0136549 A1
140
/
AUTOMATICALLY ANALYZE LUMINANCE PARAMETERS
FROM A DIGITAL IMAGE OF A BIOLOGICAL TISSUE SAMPLE
TO WHICH A MARKER DYE HAS BEEN APPLIED TO
/\
DETERMINE ONE OR MORE AREAS OF INTEREST IN THE
142
BIOLOGICAL TISSUE SAMPLE WITHIN THE DIGITAL IMAGE.
I
AUTOMATICALLY ADJUST LUMINANCE PARAMETERS
WITHIN THE ONE OR MORE DETERMINED AREAS OF
INTEREST WITHIN THE DIGITAL IMAGE TO CREATE ONE OR
MORE ADJUSTED AREAS OF INTEREST
.144
I
AUTOMATICALLY IDENTIFY PLURAL EPITHELIAL AREAS IN
THE ONE OR MORE ADJUSTED AREAS OF INTEREST
WITHIN THE DIGITAL IMAGE FOR CELL CLASSIFICATION
USING CELL MEMBRANE ANALYSIS
146
I
AUTOMATICALLY IDENTIFY PLURAL CELL NUCLEI WITH
THE PLURAL EPITHELIAL AREAS IN THE ONE OR MORE
ADJUSTED AREAS OF INTEREST WITHIN THE DIGITAL
IMAGE
A
148
I
AUTOMATICALLY IDENTIFY PLURAL CELL MEMBRANES IN
THE ONE OR MORE ADJUSTED AREAS OF INTEREST
/'\
WITHIN THE DIGITAL IMAGE
"50
AUTOMATICALLY CLASSIFY PLURAL IDENTIFIED CELL
NUCLEI WITH A PRE-DETERMINED CLASSIFICATION
SCHEME
/\
152
AUTOMATICALLY COMPLETE A MEDICAL DIAGNOSIS
GRADE BASED ON THE CLASSIFIED CELL NUCLEI
_
154
Patent Application Publication Jun. 23, 2005 Sheet 14 0f 29
FIG. 16A
US 2005/0136549 A1
156
FIG. 16B 158
160
162
FIG. 1 6 C
164
Patent Application Publication Jun. 23, 2005 Sheet 15 0f 29
US 2005/0136549 A1
FIG. 16E
176
Patent Application Publication Jun. 23, 2005 Sheet 16 0f 29
US 2005/0136549 A1
FIG. 17
180
AUTOMATICALLY IDENTIFY PLURAL CELL NUCLEI IN ONE
OR MORE ADJUSTED AREAS OF INTEREST WITHIN A
DIGITAL IMAGE
/\
182
AUTOMATICALLY IDENTIFY PLURAL CELL CHROMATIN
PATTERNS IN THE oNE OR MORE ADJUSTED AREAS OF
INTEREST
/\
184
T
AUTOMATICALLY IDENTIFY PLURAL CELL NUCLEOLUS
PATTERNS IN THE ONE OR MORE ADJUSTED AREAS OF
/1\86
INTEREST
T
AUTOMATICALLY IDENTIFY PLURAL MITOTIC CELL
PATTERNS IN THE ONE OR MORE ADJUSTED AREAS OF
/1\88
INTEREST
T
AUTOMATICALLY IDENTIFY PLURAL EPITHELIAL AREAS IN
THE oNE OR MORE ADJUSTED AREAS OF INTEREST
/1‘90
AUTOMATICALLY PLURAL STROMAL AREAS IN THE ONE
OR MORE ADJUSTED+AREAS OF INTEREST
192
AUTOMATICALLY IDENTIFY PLURAL TUBULE AREAS IN THE
ONE OR MORE ADJUSTED AREAS OF INTEREST
j
194
Patent Application Publication Jun. 23, 2005 Sheet 17 0f 29
US 2005/0136549 A1
FIG. 18
196
AUTOMATICALLY COMPUTE IMAGE STATISTICS IN ONE OR
MORE ADJUSTED AREAS OF INTEREST
F\
t
198
AUTOMATICALLY SEGMENT INDIVIDUAL PIXELS
BELONGING TO PLURAL CELL NUCLEI BY THRESHOLDING
IN THE ONE OR MORE ADJUSTED AREAS OF INTEREST
200
I
AUTOMATICALLY IDENTIFY PLURAL SEGMENTED PIXELS
BASED ON CONNECTIVITY
I
202
I
AUTOMATICALLY DETECT PLURAL CELL NUCLEI BASED ON
PRE-DETERMINED LIMITS ON A SIZE OF A NUCLEUS
’_\
204
AUTOMATICALLY IDENTIFY BOUNDARY PIXELS OF
INDIVIDUAL NUCLEI
I
\
206
AUTOMATICALLY CLASSIFY PLURAL CELL NUCLEI USING A
PRE-DEFINED CLASSIFICATION SCHEME
\
208
Patent Application Publication Jun. 23, 2005 Sheet 18 0f 29
US 2005/0136549 A1
FIG. 19A
210
/
NUCLEUS
NO
SIZE <
FIRST LIMIT?
YES
V
AUTOMATICALLY DELETE NUCLEUS AS IT
/~
REPRESENTS A SMALL NON-SPECIFIC BIOLOGICAL
214
CQMPMM
216
NUCLEUS
TO A
FIG. 198
SIZE > FIRST LIMIT
AND < SECOND
N
LIMIT?
/ 218
YES
v
AUTOMATICALLY CLASSIFY NUCLEUS AS A
NUCLEUS OF A FIRST TYPE
@
Patent Application Publication Jun. 23, 2005 Sheet 19 0f 29
US 2005/0136549 A1
FIG. 19B
SIZE > SECOND
IMIT AND < THIRD
YES
V
AUTOMATICALLY CLASSIFY NUCLEUS AS A
NUCLEUS OF A SECOND TYPE
224
NUCLEUS
SIZE > THIRD LIMIT
AND < FOURTH
NO
LIMIT?
YES
/' 226
'
AUTOMATICALLY CLASSIFY NUCLEUS AS A
NUCLEUS OF A THIRD TYPE
/
228
AUTOMATICALLY DELETE NUCLEUS AS IT
REPRESENTS A LARGE NON-SPECIFIC BIOLOGICAL
COMPONENT
/\
222