DBM MRMC 2.1
BETA VERSION 2
WINDOWS™ Operating System
(September 2006)
Kevin S. Berbaum, Ph.D.
Medical Image Perception Laboratory
Department of Radiology
The University of Iowa
e-mail: [email protected]
Charles E. Metz, Ph.D.
Kurt Rossmann Laboratories
For Radiologic Image Research
Department of Radiology
The University of Chicago
e-mail: [email protected]
Lorenzo L. Pesce, Ph.D.
Kurt Rossmann Laboratories
For Radiologic Image Research
Department of Radiology
The University of Chicago
Kevin M. Schartz, Ph.D., M.C.S.
Medical Image Perception Laboratory
Department of Radiology
The University of Iowa
DBM MRMC User's Guide (Windows OS)
Page 1 of 45
Contents
Contents ............................................. 2
System Requirements & Notes .......................... 4
Disclaimer & licensing agreement ............................4
System Requirements .........................................4
Where to find this software .................................5
How to install (and uninstall) the software .................5
Request for feedback ........................................6
An Overview of DBM MRMC2.X ............................ 7
Purposes of DBM MRMC.........................................7
Warnings and notes about the DBM MRMC algorithm..............8
Statistical models.....................................................8
ROC curve models and indices...........................................8
Sketch of the various steps of the DBM MRMC algorithm..................9
Acknowledgments ............................................10
To Run DBM MRMC ..................................... 11
To start the program .......................................11
Figure 1. Program main window. ............................12
Options available at the start .............................12
Execution of an MRMC analysis ..............................13
Open and read the input file..........................................13
Select the analysis options...........................................14
Recategorization options..............................................15
Running the analysis..................................................15
Reading result of analysis............................................16
Input file description .............................. 18
Input data .................................................18
Input file types ...........................................18
LABMRMC input file format description ......................18
To create an input file using Microsoft Word™ ........................21
To create an input file using Microsoft Excel™ .......................23
Legacy MRMC input file format description ..................25
Modern MRMC input file format description ..................30
Output files description ............................ 31
List of output files with description of their content .....31
Description of program output ..............................32
DBM MRMC User's Guide (Windows OS)
Page 2 of 45
Definition of ROC Terms ............................. 33
Actually-positive and actually-negative ....................33
TPF, FPF, TP, FP, TN, FN ...................................33
Confidence ratings and continuous ratings and categorical
data...........................................................33
Area, Area under the curve, Az ..............................34
Leave-None-Out, Leave-One-Out and pseudovalues .............34
Conventional binormal model (RSCORE model) .................34
Proper binormal model (PROPROC or PROPROC2 model) ..........35
LABROC4/LABROC5 categorization .............................36
APPENDIX ............................................ 38
Using the Recategorization menu option................................38
References .......................................... 45
DBM MRMC User's Guide (Windows OS)
Page 3 of 45
System Requirements & Notes
This version of DBM MRMC has been tested on the 32-bit versions of Microsoft Windows
2000™ and XP™. The software has not been tested on Windows 95, 98, NT, or Vista. We plan
to release versions for other operating systems in the future.
In this document we will be using the expression “DBM MRMC” to refer to the version of DBM
MRMC downloaded with this user’s guide. Occasionally we will specify which version of this
software this guide actually refers to. Most comments apply to all versions.
Disclaimer & licensing agreement
The use of DBM MRMC is constrained by the licensing agreement that users are required to
acknowledge before downloading the software. Using the software assumes acknowledgement of
that agreement. We ask that you do not distribute this software to others. However, any copying
or distribution of this software implies that the person distributing it assumes full responsibility
for informing the recipients of the licensing agreement with The University of Chicago and
University of Iowa. Modification of the software should be indicated in any modified version of
this software. Failure to acknowledge the source of this software is breach of copyright law.
By using this software to analyze data, you agree that any publications based on your analysis
will cite the appropriate DBM MRMC references listed at the bottom of the ANOVA output that
is produced. Also reference the Medical Image Perception Laboratory website
(http://perception.radiology.uiowa.edu) and the Kurt Rossmann Laboratories for Radiologic
Image Research website (http://xray.bsd.uchicago.edu/krl/), but do NOT reference this user’s
manual.
Although this software has been carefully tested, neither The University of Chicago, The
University of Iowa nor any of the individuals (present or past) who participated in the
development or testing of this software and user’s guide are responsible for any errors or for any
damages that may result from use of the software.
Inquiries or comments concerning this program should be directed to the addresses on the cover
page
System Requirements
•
•
•
A PC equipped with a Pentium or later processor.
Windows 2000™ or Windows XP™
The program requires the Microsoft .NET Framework Version 1.1 to be
installed on the system. You can visit
DBM MRMC User's Guide (Windows OS)
Page 4 of 45
http://msdn.microsoft.com/netframework/downloads/howtoget.aspx to check
whether you have the .NET framework installed and to view instructions for
downloading via Windows Update.
Where to find this software
DBM MRMC 2.X is available from the following web sites:
• http://perception.radiology.uiowa.edu/
Follow the link under the “software” tab on top of the page
• http://xray.bsd.uchicago.edu/krl/
Follow the link at the “software” tab on the right of the page.
In case of downloading problems, use the contacts provided on the cover page.
How to install (and uninstall) the software
•
•
•
•
•
•
•
•
Connect to either of the web sites mentioned in the previous section
Follow the online instructions to download the DBM MRMC zip archive to your
computer. When downloading the file, the user should select the “Save” option rather
than the “Open” option because it is necessary to save the zip file to the user’s hard drive
for proper extraction.
Unzip the DBM MRMC archive (select option to preserve folder structure).
Select the “DBM” folder from the location where you unzipped the archive
Select the “DBM MRMC 2.1 beta2 Installed” folder
The zip file contains three files: setup.exe, setup.ini, and
DBM_MRMC_Beta_Setup.msi. Double click the icon “setup.exe”
Follow the instructions of the installation wizard
The installer will install the program to the Program Files folder on the user’s machine
and create a Start menu entry and desktop shortcut for the program. An uninstaller for
the program is also created in the Add/Remove program applet in the Control Panel.
Note1: We recommend installing the program in the installation folders suggested during the
installation process itself. In case other installation schemes should be followed we won’t be able
to help you in case the installation should fail.
Note2: The standard installation will create an icon on the desktop as well as a Start menu entry.
The program can be uninstalled using the “Add or Remove Programs” applet in the Control
Panel.
Note3: The standard installation is to C:\Program Files\DBM Software\DBM MRMC 2.1 beta
(from now on called the DBM MRMC folder), in this directory the DLLs (dynamically linked
library) for all the subunits of DBM MRMC are also available. The DLLs can be called by
external programs such as SAS and used independently. This topic is addressed in more detail on
the Medical Image Perception Laboratory website (http://perception.radiology.uiowa.edu).
DBM MRMC User's Guide (Windows OS)
Page 5 of 45
Disclaimer: We cannot assume any responsibility for problems created during the installation or
un-installation process
Request for feedback
DBM MRMC 2.1 BETA2 is a Beta version of this software. Although we have tested this
software extensively and fixed all of the bugs discovered, there is no guarantee that we
have found every bug. If you discover any possible errors or anomalies in the software,
please contact us immediately so that we may fix the problem. Our address appears on the
cover page of this guide.
DBM MRMC User's Guide (Windows OS)
Page 6 of 45
An Overview of DBM MRMC2.X
DBM MRMC is designed to perform an analysis of variance when both reader and case variation
are relevant to calculate the statistical significance of the differences between different
treatments (diagnostic tests, or modalities). It will also perform an analysis with the readers or
the cases as fixed factors, if the user requires it. It should be noted that the user must determine
which ANOVA option is appropriate for her or his experimental design—all options are not
appropriate for all experimental designs. It uses conventional ROC analysis (1) on both
continuously-distributed and ordinal category data (e.g., “confidence-rating”) diagnostic test
results. There are no predetermined limits on the number of readers, treatments/modalities, or
cases.
Purposes of DBM MRMC
DBM MRMC is designed to determine the statistical significance of difference between ROC
indices when the performance of a diagnostic device is affected both by the cases analyzed (e.g.,
patient, specimen or sample) and by the observer (e.g., technician or physician)
The required data are paired test results from actually-negative (e.g., “normal” or “noise”)
cases and actually-positive (e.g., “diseased” or “signal”) cases. This means that every reader will
read every case using every modality. This is a fully-crossed design.
The description of the ANOVA model can be found in Dorfman et al. (2). However changes
have been introduced to the original model by Hillis et al. (3, 4) and those changes are
implemented in the current version of the code. A familiarity with these three articles is assumed
in the remainder of this document.
DBM MRMC can compute the ANOVA analysis under the following models:
A. Treating both readers and cases as random samples
B. Treating only cases as random samples
C. Treating only readers as random samples
It should be remembered that only option A allows the results to be generalized to the population
of readers and cases from which the used cases and readers are sampled. Option B assumes the
user only wishes to generalize to the population of cases, and option C assumes the user only
wishes to generalize to the population of readers.
DBM MRMC User's Guide (Windows OS)
Page 7 of 45
Warnings and notes about the DBM MRMC algorithm
Note:
The input data must not include more than one test result value from each readertreatment-case combination. If multiple test results from any reader-treatment-case
combination are included in the input, the program will overestimate the statistical
significance of any apparent difference between the ROC curves, thereby invalidating
the statistical test. When multiple test results are available from each reader-treatmentcase combination (for example, from replication in each condition) please contact the
people on the cover of this document or use a different analysis.
Statistical models
The available statistical models were already discussed in the previous section and references
therein.
ROC curve models and indices
DBM MRMC computes the ROC indices used in the statistical analysis using the following
models for the ROC data:
Two Proper models (to be defined below):
•
Proper binormal model (from now on called PROPROC2 because it is the second
version of the proper binormal model implemented at The University of Chicago)
•
Contaminated binormal model (from now on called CBM2, because it is the second
version of the proper binormal model implemented at The University of Iowa)
One “conventional” model:
•
Conventional binormal model in the RSCORE 4.7 implementation (referred to as
RSCORE in this documentation)
One empirical or non-parametric method:
•
Trapezoidal/Wilcoxon, non-parametric estimation of ROC indices
The trapezoidal or Wilcoxon model does not make any distributional assumption with regards to
the positive and negative cases--this is why it is non-parametric. Details about this model can be
found in Delong et al. (5). The other three models are semi-parametric, in the sense monotonic
transformations of the test result values (the quantity measured in the experiment) are assumed to
have normal distributions, one for the negative and one for the positive cases (actually two for
the contaminated model), even if the decisions about the state of cases are not made in the same
way. PROPROC2 assumes that decisions are made using likelihood ratios, i.e., the decisions
about an unknown case are based upon the relative likelihood of being positive for that specific
value (as an ideal observer would do) (6). CBM2, assumes that part of the positive cases are
indistinguishable from the negative cases (the contamination), and part is separated (7).
RSCORE assumes that the true ROC curve for each reader-treatment combination plots as a
DBM MRMC User's Guide (Windows OS)
Page 8 of 45
straight line on "normal-deviate" axes, or equivalently, that the input data follow normal
distributions for both the actually negative and actually positive fractions of the population after
some unknown monotonic transformation. ROC curves measured in a broad variety of fields
have shapes compatible with these assumptions. The assumptions are often acceptable even
when the raw data have multimodal and/or skewed distributions because it is equivalent to
assume that some variable functionally related to the one measured in the experiment (the test
result value) is normal for both conditional distributions (actually negative and actually positive),
respectively. The proper models have the great advantage of not producing curves that have
change in convexity, which is considered incompatible with medical evidence (8).
A number of indices are available for the analysis:
•
Area under the curve (AUC)
•
Partial area under the curve between two value of FPF or two values of TPF (to be
defined)
•
Sensitivity for a value of specificity
•
Specificity for a value of sensitivity
For a description of the meaning of the different indices the users are referred to the book of
Pepe (1) or other papers or books on medical diagnostics.
Note that not all indices are available for all models for different versions of the software. Refer
to the copy of the software you have or to the web site for the most complete version. In general
more recent versions have more features.
Note: any of the indices can be used for the analysis. If another index is desired, the analysis
needs to be repeated. It should be noted that repeating the analysis for multiple indices and
“looking” for the best “statistical significance” technically violates the assumptions of the
statistical analysis and invalidates the statistical tests.
Sketch of the various steps of the DBM MRMC algorithm
Data Categorization step. In a preliminary step, DBM MRMC automatically categorizes
continuously-distributed input data for each reader-treatment combination in an attempt to
produce a useful spread of operating points on each ROC curve. The program either follows the
LABROC4 algorithm that produces no loss of ROC information (9) or a binning algorithm. If
the data is categorical, empty categories are removed and identical categories are collapsed. See
section “Definition of ROC terms” for details. This again does not affect the estimation of any
ROC index, whether done non-parametrically or using maximum likelihood estimation (9). This
collapsing is done to render the computation more stable, and, again, does not affect the results.
Pseudovalue creation step. Each of the marginal categorical data sets created by the
categorization for each reader-treatment combination is then used independently to estimate the
desired ROC index. These parameter estimates are called the Leave-None-Out estimates. The
program then leaves each case out of the reader-treatment sample in turn and computes new
estimates. These estimates are called the Leave-One-Out estimates [one for each case]. Next,
the Leave-None-Out and the Leave-One-Out estimates of the ROC index are combined to create
“pseudovalue” estimates for each reader-treatment-case combination (2, 4). This step created a
matrix, with data entries (pseudovalues) for each reader-treatment-case combination.
DBM MRMC User's Guide (Windows OS)
Page 9 of 45
ANOVA step. DBM MRMC takes these “pseudovalue” estimates and computes, using ANOVA
methods (2-4) the statistical significance of the differences among treatments, etc.
⇒
Note: After each of the cases is removed to compute the Leave-One-Out, the
categorization procedure may be repeated to insure the numerical stability of the
algorithm. The comments about categorization made before apply also to this step.
Acknowledgments
This guide was compiled by Lorenzo Pesce and Kevin Schartz.
The algorithm employed by DBM MRMC was designed by Donald Dorfman, Kevin
Berbaum, Charles Metz and Steve Hillis. The program itself was written by Lorenzo Pesce
and Kevin Schartz, who also designed part of the algorithm and tested it. The code is an
almost completely rewritten version of parts originally written by Benjamin A. Herman and
Hatem Abu-Dagga, with some code taken from earlier programs written by Jong-Her Shen,
Helen Kronman and Xiaochuan Pan.
Development of this software was supported by the National Institute of Health under Grant
RO1 EB000863 (Kevin S. Berbaum, P.I.).
By using this software to analyze data, you agree that any publications based on your analysis
will cite the appropriate DBM MRMC references listed at the bottom of the ANOVA output that
is produced and also reference the Medical Image Perception Laboratory website
(http://perception.radiology.uiowa.edu) and the Kurt Rossmann Laboratories for Radiologic
Image Research website (http://xray.bsd.uchicago.edu/krl/).
DBM MRMC User's Guide (Windows OS)
Page 10 of 45
To Run DBM MRMC
To start the program
⇒
⇒
Start the DBM MRMC application by double-clicking on its icon (Be sure to have open the
folder in which the icon is located. Standard installations have the icon on the Desktop.).
The icon is called “DBM MRMC 2.1 beta.”
Alternatively it can be started from the “Start” menu shortcut
A brief splash screen will appear to indicate the release number of the edition of the program
installed.
The program main window (see Figure 1) is made of a menu bar and program page, like most
Windows programs. The main program form contains a list of the limitations of the version
installed. They should be read and understood carefully.
Note: Throughout the program execution only the tabs or options that are colored in black are
available, the other ones cannot be selected. Note that this can change during the different phases
of execution to serve as a guide as to what options are appropriate during the current phase of
execution. This can also change in different versions of the program as more options become
available.
Note: After having concluded the analysis of a dataset, in the beta version it is necessary to
restart the program to analyze another dataset .
DBM MRMC User's Guide (Windows OS)
Page 11 of 45
Figure 1. Program main window.
Options available at the start
At the beginning the only available tabs on the menu bar are the “File menu and the “Help”
menu.
The “Help” tab has two available options:
• “About” option which describes the current version of the program, and
• “Users Guide” option that opens the user’s guide (this one or a run time user’s guide,
depending upon the version).
The “File” tab, has three available options:
• “Open” is the standard Windows™ pop up window that allows to select the input file
• “User Options” selects a window that allow the user to select:
o A default data folder where the program will look for data files
o A preferred data file type (data file type will be defined in the following sections)
• “Exit” to exit the program
DBM MRMC User's Guide (Windows OS)
Page 12 of 45
Execution of an MRMC analysis
Once the program has started these are the steps to be followed to run an MRMC analysis:
Open and read the input file.
Select the “Open” option of the “File” tab, and use the pop up window to select the desired input
file. If a test run of the program is desired, we recommend the user to click the “File” tab and
select the DBM MRMC folder (it is the one selected by default in a standard installation). In the
folder an input file can be selected, for example “SAMPLE1.lrc”, which has the lrc format (the
old LABMRMC format, described below). Description on how to make and name input files are
provided in the following sections. The program does not accept keyboard input. After this stage
from the “View” tab the input file can be viewed in another window.
DBM MRMC User's Guide (Windows OS)
Page 13 of 45
Select the analysis options
Select the “Analysis options” option from the “Run” tab (which was activated by reading an
input file”).
The following screen will appear:
On the left side, on top, the different options available for the ROC model are listed. Note that
some options might be set in gray. This means that those models are unavailable in this software
version. The models have been described in the Overview section of this user’s guide, where the
appropriate references are reported. Only one model can and must be selected.
On right side, on top, the list of the available indices is described. The indices available for the
specific ROC models will appear in black, the other ones will be gray. As already stated, indices
might no be available for all ROC models at the current time.
• The “Area” is the area under the curve (AUC), also known as Az for the conventional
binormal model, here implemented as RSCORE. It is an average index and corresponds
DBM MRMC User's Guide (Windows OS)
Page 14 of 45
•
•
•
•
to the probability of actually-positive cases to have a signal larger than actually-negative
cases.
The “Partial area” corresponds to the area under the curve computed between either two
values of FP (or FPF) or two value of TP (or TPF). Any two values between 0 and 1 can
be chosen, as long as the first value is smaller than the second. Note that the partial area
between 0 and 1 is the same as the area under the curve.
“Sensitivity at specificity of” represents the estimate value of sensitivity when the
specificity is set to a specific value. All curves will have all possible values of sensitivity
or specificity, but they will map to each other differently. Note that FP or FPF = 1 –
specificity and TP or TPF = sensitivity.
“Specificity at sensitivity of” represents the estimate value of specificity when the
sensitivity is set to a specific value.
The cutpoints or threshold analysis is not fully developed yet and is not available.
The bottom part of the window allows the user to select the type of ANOVA model desired:
A. Analysis treating both readers and cases as random samples
B. Analysis treating only cases as random samples
C. Analysis treating only readers as random samples
The first option (A) is the traditional MRMC analysis, the other two assumes that either the
readers used are the only readers relevant to this analysis (B) or the cases used are the only cases
relevant for this analysis (C); the latter two options do not allow inferences to be drawn for the
whole population of readers and cases.
Recategorization options
Some models (CBM2, RSCORE) require recategorization if the data set contains more than 20
categories, but other models don’t (PROPROC2, WILCOXON). However, for challenging
datasets it might happen that the algorithms do not converge unless the number of categories is
reduced. Moreover, when extremely large datasets with many categories are used, it can be
prohibitively time-consuming to run an analysis without reducing the number of categories, i.e.,
recategorizing , because the computation time is affected mostly by the number of categories
used.
There are various options of recategorization, See the “Definition of ROC terms section for
details”. Recategorization is unnecessary in many situations. See a description of
recategorization in the Appendix for further details.
Running the analysis
The algorithm needs first to generate the pseudovalues to run the ANOVA. At this stage there
are two options from the “Run” tab:
A. “Run All”, will automatically compute the pseudovalues and do the ANOVA calculations
without further user interaction. If recategorization is necessary for the data, the
Recategorization dialog will be displayed (see Appendix). The Run All option is
appropriate for most users.
B. “Generate Pseudovalues”. If this option is selected, the pseudovalues are computed, but
the ANOVA is not performed. This option would be appropriate if the user only wished
to generate pseudovalues to be analyzed with separate software. Note that after the
DBM MRMC User's Guide (Windows OS)
Page 15 of 45
“Generate Pseudovalues” option a number of output files will be available for inspection
or analysis. This is useful if there are problems in the calculation or if a different analysis
is desired and so only the pseudovalues values are required. After this stage the “Run
ANOVA” option of the “Run” tab needs to be selected to finish the analysis. The
ANOVA output file will be written after the completion of the ANOVA computations.
Dialogs will appear to display progress of the computations.
Reading result of analysis
Under the tab “View” a number of options are listed, those are the different output files produced
by MRMC. Selecting a file will prompt the program to open a specific application that will
display the file. The files are:
•
“Input”, shows in Notepad™ the input file used for this calculation. This can be used to
verify if indeed the correct input file was selected.
•
“Pooled” displays the results of Leave-None-Out calculations (see overview section and
references therein), opened in Excel™. Description:
o First column, the treatment (number)
o Second column, the reader (number)
o Third column: the number of effective categories used (that is if the user used 4
categories out of 5, this will show a 4)
o Fourth column, the value of the index chosen
o Fifth and sixth column, the parameters of the model (if semi-parametric)
otherwise they will not be present.
•
“Pseudovalues” the list of the pseudovalues in list form, by column, opened in Excel™:
“R” = Reader number
“T” = Treatment number
“C” = Case number
“P” = Value of pseudovalue
“IC” = Initial Category (prior to any recategorization) to which this pseudovalue
corresponds (all pseudovalues from the same category have the same value)
o “RC” = Recategorized Category (after any recategorization) to which this
pseudovalue corresponds (all pseudovalues from the same cateogory have the
same value)
o “CC” = Collapsed Category (after application of LABROC4 or other algorithm to
eliminate empty/unnecessary categories) to which this pseudovalue corresponds
(all pseudovalues from the same category have the same value)
o
o
o
o
o
•
“ROC Plot Info”, information about the ROC plot. Opened in Excel™. This can be used
to plot the curves for each reader treatment combination. The file is divided in blocks that
follow one another vertically. Each block is defined by the header “Reader x Treatment
y” and tells that the following data regards readers x treatment y.
o The first vertical sequence of numbers (under the columns FPF and OBS)
represents the empirical operating points as determined by the input data.
o The second vertical sequence of numbers (under the columns FPF and THEOR)
represents the theoretical operating points. The different operating points
correspond to “cutoffs” on the ROC curve. If the cutoffs estimated for the
empirical operating points are used on the estimated curve one obtains the
DBM MRMC User's Guide (Windows OS)
Page 16 of 45
theoretical operating points. They are the place where the operating points would
be for an infinitely large dataset if the population curve is the estimated curve. It
is absent for the non-parametric methods.
o The third vertical sequence of numbers (under the columns FPF and TPF)
represents a sequence of 200 points on the estimated ROC curve. They can be
used to plot the estimated ROC curve. It is absent for the non-parametric methods.
•
“ANOVA”, Notepad™ opens a file that contains the information about the ANOVA
calculations. The file itself is amply commented, and the terms that appear in the file are
described in the references (2-4), which should be referenced (together with the web
sites) when using this program for research publications.
DBM MRMC User's Guide (Windows OS)
Page 17 of 45
Input file description
Here we describe the different input files formats accepted by DBM MRMC and then shortly
describe how to prepare input files using conventional text editing or spreadsheet programs.
These last instructions are not meant by any means to be exhaustive or even appropriate for the
specific kind of data and data collection used by all possible users, but only to provide some
guidance and help determine the source of potential I/O problems.
Input data
Data is usually defined as continuous, categorical or ordinal data and confidence data.
Continuous data, and all data that is numeric, is entered in the input file as it is: the data can be
entered directly following the instructions of the following sections. Confidence data, if
expressed in terms of language, e.g., “signal possibly present” (see “Definition of ROC terms”
section), has first to be transformed into numerical values, usually integers. Of course care must
be put to give larger values to categories that express higher probability of containing the signal.
Input file types
The different types of files available in the version installed can be viewed by selecting File ->
Open and clicking the “Files of type:” drop-down filter. Here is a description of those input files:
• LABMRMC files should have a file extension of .lrc or .txt in order to be recognized by
the program.
• Files in the legacy MRMC file format (e.g., for MRMC 1.70) should have the .in or .inp
extension.
• The extension of .fsp is used to indicate a file in a revised MRMC format where all
fields are four characters wide (I4 format for Fortran programmers).
• The extension of .csv is used to indicate a file in a revised MRMC format where all
fields are separated by commas rather than appearing in particular columns (commadelimited rather than a fixed format)
If a file does not have one of these extensions or has the wrong extension for its type, it will
not be properly read, and you will receive an error message (if the selected file is a binary
rather than text file, the program may crash). If you encounter a problem reading a file in
the beta version of the program, you will need to restart the program before attempting to read
another file because the File -> Open option will become disabled.
Some sample data files are available for download from the websites
LABMRMC input file format description
(Please see example on page on page 22 for input files created using Microsoft Word, or
example on page 23 for input files created using Microsoft Excel)
Numbers represent input lines or groups of input lines, whereas bullets represent the description
of those input lines. An example of a part of a LABMRMC file is provided in figure 2 of this
section. An example input file is also provided with the installation of DBM MRMC, it is the file
SAMPLE1.lrc.
1. A free-text description for the file (up to 60 characters, including any leading blanks).
DBM MRMC User's Guide (Windows OS)
Page 18 of 45
ƒ
This description allows you to identify easily the type of data stored in the file.
For example, if the current file contains information on a 10 observer study of 4
different mammographic CAD techniques, then this line might be:
“2006 CAD comparison in mammo, 10 readers, 4 treatments”
2. The name of the particular reader whose data you are about to enter (starting with the first
one).
3. On one line, enter the names of all the treatments
ƒ
Each treatment name must be enclosed in quotes (“”) and be no more than 12
characters long.
ƒ
DBM-MRMC 2.1 and later does not have limitations on the number of
treatments used.
4. On one line enter an alphabetic code word (“small” or “large” or simply “s” or “l”) for each
treatment, separated by one or more blank spaces, to indicate that smaller or larger test
results, respectively, are associated with stronger evidence of positivity (e.g., “signal”,
“disease” or “abnormality”).
ƒ
DBM-MRMC 2.1 reads only the first character of each of these code words.
5. A sequence of test-result values for actually-negative (e.g., “noise” or “normal”) cases.
ƒ
On a line for each actually-negative case, enter the test result for treatment 1, one
or more blank spaces, the test result for treatment 2, one or more blank spaces,
the test result for treatment 3, etc. Optionally, these test results can be followed
by one or more blank spaces and then a brief free-text description of the case.
The description must follow the values of all modalities (i.e., be at the end of the
line).
ƒ
It is absolutely essential that the cases appear in the input data in exactly the same
order for each reader.
ƒ
The program currently requires all test results values for each condition
ƒ
Each test-result value can include up to 6 digits to the left of the decimal point
(in addition to a “+” or “-” sign) and up to 6 digits to the right.
ƒ
DBM MRMC 2.1 and later does not have limits on the number of actuallynegative cases that can be entered.
ƒ
Input of actually-negative cases must be terminated by a final line containing an
asterisk (*) as its first character.
6. A sequence of test-result values for actually-positive (e.g., “signal”, “diseased” or
“abnormal”) cases.
ƒ
The input format is the same as that for actually-negative cases.
DBM MRMC User's Guide (Windows OS)
Page 19 of 45
ƒ
DBM MRMC 2.1 and later does not have limits on the number of actuallypositive cases that can be entered.
ƒ
Again, the input sequence must be terminated by a final line of free text
containing an asterisk (*) as its first character and a “Enter” (“Return”) as its
last character.
7. Repeat steps 2, 5 and 6 for the remaining readers. DBM MRMC 2.1 and later does
not have limits on the number of readers in a data set. End the dataset with a pound
sign (#) and a “Enter” (“Return”).
Acceptable file name extensions for the LABMRMC format are “.txt” and “.lrc”. We
recommend to use “.lrc” because the “.txt” might be restricted in future versions. When making
the input files with other programs that will save them as “.txt”, it is enough to change their
names after the file is finished.
DBM MRMC User's Guide (Windows OS)
Page 20 of 45
Using other programs to create input files
Many programs can be used to create the input files as described above. However, it is
essential that these programs save the input file in a text-only (word processor) or
formatted-text (spreadsheet) format. Although we shall mention only Microsoft Word™
and Excel™ in the examples below, most other equivalent programs will work in a
similar fashion.
To create an input file using Microsoft Word™
ƒ
Open Microsoft Word™.
ƒ
Open a new document by selecting “New...” from the “File” Menu.
ƒ
Enter your data into the file according to the one of the file formats described above.
ƒ
Save the file:
— select “Save As...” from the “File” Menu;
— In the “Save File as Type” pop-up menu choose “MS-DOS TEXT with Layout”
or “Text Only” (the names will vary depending on the version of Windows™ and
Word™ that you are using; in the worst case keep trying until one works);
— type the name you want to save the file as; and
— click on the “Save” button or hit the “Enter” (“Return”) key to save the file.
Figure 1. Save for input file created with Microsoft Windows™. Here the name of the folder is
“rockit”, and the name of the file is “sample in.txt”
DBM MRMC User's Guide (Windows OS)
Page 21 of 45
Figure 2. . Example of Microsoft Word™ input file (LABMRMC input file)
DBM MRMC User's Guide (Windows OS)
Page 22 of 45
To create an input file using Microsoft Excel™
ƒ
Open Microsoft Excel.
ƒ
Open a new document by selecting “New...” from the “File” Menu.
ƒ
Enter your data into the file according to one of the file formats described above:
— Enter each number in its own column.
⇒
Note for users using the LABMRMC format: Excel does not automatically enclose
text in quotes. Therefore you must enclose the condition descriptions in quotes and
always save as “formatted text” or “TEXT(MS-DOS)” or some similar name,
depending upon the version of Windows™ and Excel™ that you are using.
— Enter each line in its own row.
ƒ
Save the file:
— select “Save As...” from the “File” Menu;
— choose “Formatted Text” or “TEXT(MS-DOS)” in the “Save File as Type”
pop-up menu (see comment at the previous point);
— type the name you want to save the file as; and
— click on the “Save” button or hit the “Enter” (“Return”) key to save the file.
Figure 3. Save for input file created with Microsoft Excel™
DBM MRMC User's Guide (Windows OS)
Page 23 of 45
Figure 4. Microsoft Excel™ input file (LABMRMC input file)
DBM MRMC User's Guide (Windows OS)
Page 24 of 45
Legacy MRMC input file format description
I. The original space-delimited MRMC input file format(*.in & *.inp extensions in DBM
MRMC):
line #
-----1
2
3
4
5
6
7
columns
------1-2
type
-----integer
3-4
1-4
1-2
1-2
1-4
1-2
2-4
5-6
7-8
.
.
.
1-2
3-4
5-6
7-8
integer
integer
integer
integer
integer
integer
integer
integer
integer
take two values 0 or 1 (0..1).
0 means high value of the confidence rating
is high category.
1 means low value of the confidence rating
is high category.
the value of the highest category (3..20).
number of normal trials.
number of treatments (2 ..).
number of readers (2 ..).
number of cases (2 ..).
conf. rating of reader 1, treatment 1, case 1.
conf. rating of reader 1, treatment 2, case 1.
conf. rating of reader 1, treatment 3, case 1.
conf. rating of reader 1, treatment 4, case 1.
integer
integer
integer
integer
conf.
conf.
conf.
conf.
rating
rating
rating
rating
of
of
of
of
reader
reader
reader
reader
1,
1,
1,
1,
treatment
treatment
treatment
treatment
1,
2,
3,
4,
case
case
case
case
2.
2.
2.
2.
.
.
.
And so on until the end of Reader 1 data. The same format is used for all the readers.
DBM MRMC requires at least 2 treatments. If you have one treatment, use one of the curvefitting routines such as PROPROC, CBM, or RSCORE.
The following is fragment of input file (the complete file is in separate file named
SAMPLE1.IN):
•1•5
••33
•2
•4
•100
•3•3
•5•3
•4•2
•4•3
•4•5
•4•5
(The "•" do not actually appear in the data)
DBM MRMC User's Guide (Windows OS)
Page 25 of 45
•4•1
•2•2
•5•5
.
.
.
.
This file contains the rating data for two treatments, four readers, and 100 cases for each reader.
Line 1:
The "1" in the first line of the data file indicates that higher rating values indicate higher
confidence. The "5" indicates that there are five category ratings in the data (1 through 5).
Line 2:
The "33" in the second line indicates that the first 33 cases for each reader are normal (noise),
and the last 67 are abnormal (noise + signal).
Line 3:
The "2" indicates that there are two treatments in the data set.
Line 4:
The "4" indicates that there are four treatments in the data set.
Line 5:
The "100" indicates that there are 100 cases per reader.
Remaining Lines:
These are the data lines. Note that the columns correspond to treatments, rows to cases, and
every 100 rows to a reader. The data for Reader 1 data comes before the data for Reader 2, etc.
II. Another legacy space-delimited version of the original MRMC input file format is as follows
(*.fsp extension in DBM MRMC):
This version of the format was created to make it easier to use a spreadsheet program such as
Excel to create the data files by saving them as space-delimited (*.prn) files.
line #
-----1
2
3
4
5
6
columns
------1-4
type
-----integer
5-8
1-4
1-4
1-4
1-4
1-4
5-8
9-12
integer
integer
integer
integer
integer
integer
integer
integer
take two values 0 or 1 (0..1).
0 means high value of the confidence rating
is high category.
1 means low value of the confidence rating
is high category.
the value of the highest category (3..20).
number of normal trials.
number of treatments (2 ..).
number of readers (2 ..).
number of cases (2 ..).
conf. rating of reader 1, treatment 1, case 1.
conf. rating of reader 1, treatment 2, case 1.
conf. rating of reader 1, treatment 3, case 1.
DBM MRMC User's Guide (Windows OS)
Page 26 of 45
7
13-16
.
.
.
1-4
5-8
9-12
13-16
integer
conf. rating of reader 1, treatment 4, case 1.
integer
integer
integer
integer
conf.
conf.
conf.
conf.
rating
rating
rating
rating
of
of
of
of
reader
reader
reader
reader
1,
1,
1,
1,
treatment
treatment
treatment
treatment
1,
2,
3,
4,
case 2.
case 2.
case.2
case 2.
.
.
.
And so on until the end of Reader 1 data. The same format is used for all the readers.
DBM MRMC requires at least 2 treatments. If you have one treatment, use one of the curvefitting routines such as PROPROC, CBM, or RSCORE.
The following is fragment of input file (the complete file is in separate file named
SAMPLE1.FSP):
•••1•••5
••33
•••2
•••4
•100
•••3•••3
•••5•••3
•••4•••2
•••4•••3
•••4•••5
•••4•••5
•••4•••1
•••2•••2
•••5•••5
(The "•" do not actually appear in the data)
.
.
.
.
This file contains the rating data for two treatments, four readers, and 100 cases for each reader.
Line 1:
The "1" in the first line of the data file indicates that higher rating values indicate higher
confidence. The "5" indicates that there are five category ratings in the data (1 through 5).
Line 2:
The "33" in the second line indicates that the first 33 cases for each reader are normal (noise),
and the last 67 are abnormal (noise + signal).
Line 3:
The "2" indicates that there are two treatments in the data set.
Line 4:
DBM MRMC User's Guide (Windows OS)
Page 27 of 45
The "4" indicates that there are four treatments in the data set.
Line 5:
The "100" indicates that there are 100 cases per reader.
Remaining Lines:
These are the data lines. Note that the columns correspond to treatments, rows to cases, and
every 100 rows to a reader. The data for Reader 1 data comes before the data for Reader 2, etc.
III. A comma-delimited version of the original MRMC input file format is as follows (*.csv
extension in DBM MRMC):
This version of the format was created to make it easier to use a spreadsheet program such as
Excel to create the data files by saving them as comma-delimited (*.csv) files. Unlike the
previous MRMC format descriptions, there is not mention of columns because the data does not
need to be in particular columns.
line #
-----1
2
3
4
5
6
7
type
-----integer
integer
integer
integer
integer
integer
integer
integer
integer
integer
take two values 0 or 1 (0..1).
0 means high value of the confidence rating
is high category.
1 means low value of the confidence rating
is high category.
the value of the highest category (3..20).
number of normal trials.
number of treatments (2 ..).
number of readers (2 ..).
number of cases (2 ..).
conf. rating of reader 1, treatment 1, case 1.
conf. rating of reader 1, treatment 2, case 1.
conf. rating of reader 1, treatment 3, case 1.
conf. rating of reader 1, treatment 4, case 1.
integer
integer
integer
integer
conf.
conf.
conf.
conf.
rating
rating
rating
rating
of
of
of
of
reader
reader
reader
reader
1,
1,
1,
1,
treatment
treatment
treatment
treatment
1,
2,
3,
4,
case 2.
case 2.
case.2
case 2.
.
.
.
And so on until the end of Reader 1 data. The same format is used for all the readers.
DBM MRMC requires at least 2 treatments. If you have one treatment, use one of the curvefitting routines such as PROPROC, CBM, or RSCORE.
The following is fragment of input file (the complete file is in separate file named
SAMPLE1.CSV):
DBM MRMC User's Guide (Windows OS)
Page 28 of 45
1,5
33
2
4
100
3,3
5,3
4,2
4,3
4,5
4,5
4,1
2,2
5,5
.
.
.
.
This file contains the rating data for two treatments, four readers, and 100 cases for each reader.
Line 1:
The "1" in the first line of the data file indicates that higher rating values indicate higher
confidence. The "5" indicates that there are five category ratings in the data (1 through 5).
Line 2:
The "33" in the second line indicates that the first 33 cases for each reader are normal (noise),
and the last 67 are abnormal (noise + signal).
Line 3:
The "2" indicates that there are two treatments in the data set.
Line 4:
The "4" indicates that there are four treatments in the data set.
Line 5:
The "100" indicates that there are 100 cases per reader.
Remaining Lines:
These are the data lines. Note that the columns correspond to treatments, rows to cases, and
every 100 rows to a reader. The data for Reader 1 data comes before the data for Reader 2, etc.
DBM MRMC User's Guide (Windows OS)
Page 29 of 45
Modern MRMC input file format description
Beginning with DBM MRMC 2.1 beta2 and later, the DBM MRMC software permits the use of
comments within MRMC-format data files (.in, .fsp, or .csv). Comment lines MUST begin with
an exclamation point (!) as the first character on the line. These comment lines can appear
anywhere within the data file and are ignored by the data parser (note that the entire line
following the “!” will be ignored”). Comment lines are shown in green in the augmented
example below. Also beginning with DBM MRMC 2.1 beta2, the DBM MRMC program also
permits the user to specify labels for readers and treatments. These label lines MUST begin with
a dollar sign ($). Following the dollar sign is the tag TITLE to refer to the title of the data set,
the tag TRT to refer to treatment labels, and the tag RDR to refer to reader labels. In later
releases of the software, these labels will be used in the output files produced. For beta2, the
labels can be present in the data file, but they will not appear in the output. Label lines are
shown in blue in the augmented example below. The sample file below is available from the
website as file “SAMPLE1_augmented.fsp”.
$TITLE: "Sample data set"
! Low # = High Confidence; 5 categories
1
5
! Normal Cases
33
! Treatments
2
$TRT: "Treatment 1" "Treatment 2"
! Readers
4
$RDR: "Reader 1" "Reader 2" "Reader 3" "Reader 4"
! Cases
100
! Tx1 Tx2
3
3
5
3
4
2
4
3
4
5
4
5
4
1
2
2
5
5
These comments and labels can appear in all three MRMC file formats.
DBM MRMC User's Guide (Windows OS)
Page 30 of 45
Output files description
The output of DBM-MRMC is organized in several types of files. These files are written to the
directory containing the input file. In some circumstances, DBM MRMC itself or some of the
DLLs it is calling may create additional log files. These files are there solely for debugging
purposes and can be ignored by most users.
List of output files with description of their content
The expression “Input_file_name” will be used to refer to the name of the input file without
extension (that is what comes after the “.”). In the following example the input file is called
SAMPLE1.lrc (and it is one of the example files available with the release), so Input_file_name
= SAMPLE1.
Model_name refers to the ROC model used, in the example PROPROC2 shortened to proproc.
“Norm” refers to the fact that pseudovalues are normalized. Only normalized pseudovalues are
used by MRMC2.1B2. See Hillis and Berbaum (4) for a description of normalized
pseudovalues. Index_used refers to the ROC index used. In the example “area” for area under
the curve
Naming scheme
Input_file_name model_name
norm Index_used anova.txt
Example name
SAMPLE1 proproc norm
area anova.txt
Input_file_name model_name
norm Index_used plot.csv
SAMPLE1 proproc norm
area plot.csv
Input_file_name model_name
norm Index_used pooled.csv
SAMPLE1 proproc norm
area pooled.csv
Input_file_name model_name
norm Index_used pseudo.csv
SAMPLE1 proproc norm
area pseudo.csv
DBM MRMC User's Guide (Windows OS)
Description
Text file. Output of the
ANOVA routines such as
inference results, confidence
intervals, components of
variance
CSV file made to be read
with a spreadsheet program
like Excel™. Contains
Leave-None-Out plotting
information. Used to plot
ROC curves.
CSV file made to be read
with a spreadsheet program
like Excel™. Contains
Leave-None-Out fitting
information, such as curve
parameters and area under
the curve
CSV file made to be read
with a spreadsheet program
like Excel™. Contains the
pseudovalue used in the
ANOVA calculations. To be
used as a check or for
additional calculations
Page 31 of 45
Description of program output
The output files were previously described in the "To Run DBM MRMC" section. The “View”
menu can be used to open the output files, or they can be opened from within Windows Explorer
(file manager).
DBM MRMC User's Guide (Windows OS)
Page 32 of 45
Definition of ROC Terms
Actually-positive and actually-negative
Actually-negative cases are the cases that have been verified to be negative. This means that the
researcher is very confident, or as confident as possible, that the cases in fact do not contain the
signal that is being looked after (e.g., cancer screening it means that the patient either is healthy
or has a disease so small that the medical community does not consider it a disease). The
verification is usually done using a different, very accurate, diagnostic tool, usually called gold
standard (in the case of cancer screening this is usually a pathology report with some follow up
period to insure that the diagnosis was in fact correct).
Actually-positive cases are the cases that have been verified to be positive. This means that the
researcher is very confident, or as confident as possible, that the cases in fact contain the signal
that is being looked after (e.g., cancer screening it means that the patient has the disease in a
form that the medical community considers clear). The verification is usually done using a
different, very accurate, diagnostic tool, usually called gold standard (in the case of cancer
screening this is usually a pathology report).
TPF, FPF, TP, FP, TN, FN
False positive fraction (FPF) or false positive (FP) or false positive rate (FPR) corresponds to the
number of actually-negative cases incorrectly diagnosed as positive by the modality under
investigation divided by the total number of actually-negative cases. Note that this value, unless
the test is binary (Yes/No) refers to a specific threshold, see Pepe’s book for details (1). The
same applies to all the following definitions.
True positive fraction (TPF) or true positive (TP) or True positive rate (TPR) corresponds to the
number of actually-positive cases correctly diagnosed as positive by the modality under
investigation divided by the total number of actually-positive cases.
True Negative (TN) is an actually-negative case correctly classified as negative.
False Negative (FN) is an actually-positive case incorrectly classified as negative.
Confidence ratings and continuous ratings and categorical data
With the terms “confidence ratings”, we mean inherently ordinal data labeled using expressions
that represent the confidence the rater has in the presence or absence of a signal, for example
ratings could be any of these categories: “signal definitely absent”, “signal probably absent”,
“signal possibly present, equivocal”, “signal probably present”, “signal definitely present".
Another example of categorical data is the BI-RADS score, used in mammography. These data
are often called categorical or discrete because there are many fewer categories that data points
and usually the difference between categories is rather sharp (even if this does not preclude the
existence of borderline cases). When analyzed usually the category labels are replaced with
integers, to replicate the ordering of the data, e.g.:
Signal definitely absent
1
Signal probably absent
2
Signal possibly absent, equivocal 3
Signal probably present
4
DBM MRMC User's Guide (Windows OS)
Page 33 of 45
Signal definitely present
5
“Continuous ratings” are for example lab tests or serum concentrations. They are expressed as
decimal figures (e.g., 3.456). Usually there are many more possible values than datapoints.
In general empirical datasets range from being defined with very few categories (2 in the case of
a sensitivity and specificity analysis) to subjective probability scales with many hundreds of
categories, to the quasi-continuous scales of lab tests or neural network outputs. It should be
noticed that apart from numerical and statistical issues (variance and bias of estimates) these data
are equivalent from the ROC point of view as long as they can be ranked unequivocally.
Area, Area under the curve, Az
The by far most commonly used index in ROC analysis is the are under the ROC curve (1), it
represents the average TPF value provided by the test. Usually it is called area under the curve
(AUC). This last term applies to any ROC curve, defined using any model, whether parametric,
semi-parametric or non-parametric. Sometimes this terms is simply replaced by “area” when the
context makes the interpretation unequivocal. The term Az technically refers only to the AUC for
the conventional binormal model (RSCORE here).
Leave-None-Out, Leave-One-Out and pseudovalues
The statistican inference made by this program is based on ANOVA. However, the data for the
ANOVA computation is produced using Tukey-Quenouille jackknife (10). Data is considered to
be fully crossed, i.e., every reader will read every case using every modality. If you select a
reader and a modality then there will be a sequence of actually-negative and actually-positive
cases, typical ROC data, from now on this will be called the reader-modality dataset.
The first step is to compute the ROC index estimate for the reader-modality dataset. This is
called the Leave-None-out, because no cases are left out of the sample, let’s call it θ. Since the
ROC index cannot be computed for a single case, one way to compute the effect of the case
contribution to the index is to compute the index taking that case out, θ(k), this is the Leave-OneOut. The contribution of that case can be estimated by “its negative” using the pseudovalue:
Yk = nθ − (n −1)θ(k )
Conventional binormal model (RSCORE model)
The term “conventional binormal model” is often used in comparisons with the “proper models”
- otherwise it might be simply called “binormal model”. Each possible case is assumed to be
associated with a value x. Usually, the variable x is not what is measured in the experiment being
analyzed, but a monotonic transformation of that value and it is called a latent variable (i.e., not
observed). In the model, x is normally distributed both for the actually-negative cases (the
distribution of x conditional to being negative is normal) and for the actually-positive cases (the
distribution of x conditional to being positive is normal). The two normal distributions have in
general different means and standard deviations: μn, σn for the actually-negative cases and μs, σs
for the actually-positive cases. The usual convention that positive-cases are more likely to have a
larger value implies that μn < μs.
Calling φ the normal distribution function, φ(μn, σn, -xt) is the fraction of actually-negative cases
with value above xt (i.e., the false positive fraction or FPF) and φ(μs, σs, -xt) the fraction of
actually-positive cases with value above xt (the true positive fraction or TPF). (it should be
DBM MRMC User's Guide (Windows OS)
Page 34 of 45
remembered that the fraction of cases above x is 1 - φ(μ, σ, x), which is the same as φ(μ, σ, -x)
because of symmetry.)
The ROC curve is produced by sweeping the value of xt and plotting φ(μs, σs, -xt) against
φ(μn, σn, -xt) in the ROC plot (TPF vs FPF) (1). The latent variable x can be assumed to have
produced the measurements (or test result values), which we are using to make the MRMC
study, after it had been subjected to a monotonic transformation and possibly some discretization
(to produce discrete values from the continuous values of the normal distribution). It is binormal,
because we are assuming that this random variable is normal both for actually-positive cases and
for actually-negative cases.
For most distributions, the two normal densities will intersect two times, which means that there
are either large values for which it is more likely to have an actually-negative case than to have
an actually-positive (because the actually-negative conditional distribution is larger there after
they intersected) or that there are smaller values for which it is more likely to have an actuallypositive case than to have an actually-negative (because the actually-positive conditional
distribution is larger there because the value is before the intersection). This will create a socalled hook. (Note that this behavior is sometimes not negligible.) A “proper model” is a
correction to this. The conventional binormal model is usually defined using two parameters: “a”
that represents the vertical intercept and “b” that represents the slope of the fitted ROC curve
when it is plotted as a straight line on “normal-deviate” axes. The first parameter is related to the
difference between the means of the two normal distributions, while the second parameter is the
ratio of their standard deviations.
Proper binormal model (PROPROC or PROPROC2 model)
The term “proper binormal model”, often used as opposed to “conventional binormal model”,
means that the ROC curve is produced by sweeping the value of likelihood ratios as determined
by a binormal distribution (6). The likelihood ratio (or better a monotonically increasing function
of it) is the latent variable of this model (see previous section for a definition of a latent
variable). Accordingly, the coordinates of the point on the ROC curve corresponding to each
threshold value has abscissa, the False Positive Fraction (FPF or FP), equal to the number of
actually negative cases with likelihood ratio above the threshold divided by the total the number
of actually negative cases, and ordinate, the True Positive Fraction (TPF or TP), equal to the
number of actually positive cases with likelihood ratio above the threshold divided by the total
the number of actually positive cases. The latent variable is assumed to have produced the
measurements (or test result values), which we are using to make the MRMC study, after it had
been subjected to a monotonic transformation, plus possibly a discretization. It is binormal
because we are assuming that the actually-negative and actually-positive densities from which
the likelihood ratio is computed are normal. The use of the likelihood ratio prevents the
formation of the so-called hooks. Other solutions to the “hooks” problem have been suggested
and some are included in this software package (CBM2). The exact meaning of the parameters
of the proper binormal model, da and c, is not intuitive. In general da is proportional (nonlinearly) to the value of the area under the curve, and c is proportional (also non-linearly) to the
DBM MRMC User's Guide (Windows OS)
Page 35 of 45
shape of the curve, i.e., how skewed it is. Refer to Metz et al. (6) for details about this models. A
paper by Pesce and Metz describing the algorithm used in DBM MRMC, PROPROC2, was
submitted to Academic Radiology (June 2006).
LABROC4/LABROC5 categorization
LABROC4 algorithm. Before maximum-likelihood estimation of the ROC parameters is
attempted, continuously-distributed input data are rank-ordered and then collapsed into truthstate runs as per the LABROC4 algorithm (9). Since ROC analysis is concerned with the ability
of an algorithm to separate actually-negative from actually-positive cases, once the set is rank
ordered the actual values of the test results can be replaced with their ranks with no loss of
information (i.e., the resulting ROC curve will be identical). Let’s make a simple example with a
12 case dataset. The dataset has 6 actually-negative (green) and 6 actually-positive (red) cases.
The following steps will be involved:
•
Let us suppose that on input we have the actually-negative cases first, followed by the
actually-positive (1st row)
•
The cases are ranked (2nd row)
•
The values are replaced with their ranks (3rd row)
•
Contiguous ranks of the same truth (either all actually-positive or all actually-negative)
are collapsed (4th row).
•
The cases with the same number are counted and those counts are the new categorical
data (following table). This table produces exactly the same ROC curve as the original
data, apart from possible numerical issues.
5.0
7.0
3.0
1.0
1.0
3.0
6.0
7.0
8.0
10.0
1.0
7.0
1.0
1.0
1.0
3.0
3.0
5.0
6.0
7.0
7.0
7.0
8.0
10.0
1
1
1
2
2
3
4
5
6
6
7
8
1
1
1
2
2
2
3
4
5
5
5
5
DBM MRMC User's Guide (Windows OS)
Page 36 of 45
LABROC4 categories (for example above)
Category index
I
II
III
IV
V
# actually-negative cases
2
3
0
1
0
# actually-positive cases
1
0
1
0
4
LABROC5 algorithm. The runs obtained from the LABROC4 algorithm (let us supposed that it
generated L categories, in previous example 5) can then reduced to K categories in an ad hoc but
empirically useful way, in the sense that it was empirically shown to produce good fits (9). Here
we describe a simplified version of the algorithm, the exact one can be found in Metz et al. (9).
Steps of the LABROC5 algorithm (starting from the previous LABROC4 points):
• Take the boundaries between points (in the example above they are between 0 and 1,
1 and 2, 2 and 3, 3 and 4). Since we move from the more positive to the less positive,
the boundaries are ordered from the largest value to the smallest (1st row)
• The boundaries are transformed to their operating points or positive and negative
fractions (2nd row)
• The two fractions for each point are summed, also called city block distance (3rd row)
• The city block distances are centered to 1 (which corresponds to a point on the -45
degrees diagonal) (4th row)
• These values are transformed into their normal distribution values (5th row)
• The “most uniform” values are selected and the categories in between are collapsed
4,3
3,2
2,1
1,0
(0, 4/6)
(1/6, 4/6)
(1/6, 5/6)
(4/6, 5/6)
4/6
5/6
1
9/6
-1/3
-1/6
0
½
+.37
+.43
.5
.69
OK
Collapse
OK
OK
LABROC5 categories (for example above)
Category index
I
II
III
IV
# actually-negative cases
2
3
1
0
# actually-positive cases
1
0
1
4
The LABROC5 points do not preserve the information of the original dataset.
DBM MRMC User's Guide (Windows OS)
Page 37 of 45
APPENDIX
Using the Recategorization menu option
Recategorization is necessary if the input data contains more than 20 categories and the CBM2
or RSCORE 4.7 curve-fitting options were selected (PROPROC2 and the Wilcoxon/trapezoidal
approach do not have the 20-category limitation). The user can manually invoke
recategorization in other situations if desired by selecting Recategorization from the Run menu.
After the Recategorization step, the Generate Pseudovalue step can be performed.
The recategorization dialog appears as follows:
DBM MRMC User's Guide (Windows OS)
Page 38 of 45
At the top, the columns to the left display the minimum value, the maximum value, and the
number of categories in the current data set. The columns to the right display the minimum
value, maximum value, and the number of categories permitted by the selected curve-fitting
algorithm. This example displays the values for a 101-category data set used with CBM2.
A message in red typeface indicates that recategorization is necessary for this data set. If
recategorization had not been necessary, a message in black typeface would indicate that
recategorization was not necessary. In the situation shown, the user would need to check the box
for “Recategorize the data” in order to proceed with analysis.
The various recategorization options would then be enabled. The options available to the user
will depend on the number of categories in the data set (e.g., if the data set has only 10
categories, the options to recategorize to 20 or 10 categories will not be enabled). A brief
description of each option follows:
DBM MRMC User's Guide (Windows OS)
Page 39 of 45
Recategorize to 20 categories (fine ends, coarse middle)
Assuming 101 categories (0 to 100 or subjective probability data), the following algorithm is
used
Original Data
Range
Recategorized
Value
0
1-4
5
6-10
11-15
16-20
21-25
26-30
31-40
41-50
51-60
61-70
71-75
76-80
81-85
86-90
91-94
95
96-99
100
DBM MRMC User's Guide (Windows OS)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Page 40 of 45
Recategorize to 20 categories (evenly distributed)
Assuming 101 categories (0 to 100 or subjective probability data), the following algorithm is
used
Original Data
Range
0-5
6-10
11-15
16-20
21-25
26-30
31-35
36-40
41-45
46-50
51-55
56-60
61-65
66-70
71-75
76-80
81-85
86-90
91-95
96-100
DBM MRMC User's Guide (Windows OS)
Recategorized
Value
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Page 41 of 45
Recategorize to 10 categories (evenly distributed)
Assuming 101 categories (0 to 100 or subjective probability data), the following algorithm is
used
Original Data
Range
Recategorized
Value
0-10
11-20
21-30
31-40
41-50
51-60
61-70
71-80
81-90
91-100
1
2
3
4
5
6
7
8
9
10
Recategorize to 5 categories (evenly distributed)
Assuming 101 categories (0 to 100 or subjective probability data), the following algorithm is
used
Original Data
Range
0-20
21-40
41-60
61-80
81-100
Recategorized
Value
1
2
3
4
5
Use LABROC4 algorithm to reduce categories
This algorithm is described in the “Definition of ROC Terms” section. NOTE: This
algorithm does not reduce the data to a predetermined number of categories, so if there are
a large number of categories in the data, it may not reduce the categories sufficiently for
RSCORE or CBM2.
DBM MRMC User's Guide (Windows OS)
Page 42 of 45
Use the Dorfman/MRMC legacy algorithm to reduce categories
Like the LABROC4 algorithm, this algorithm does not reduce the data to a predetermined
number of categories. As such, it may not be appropriate for data recategorization for
RSCORE and CBM2 if the number of categories in the data set greatly exceeds 20 (e.g., 101category or subjective probability data). A description of the algorithm follows.
The algorithm makes two passes over the categorical data matrix that is created based on the
input data:
1. The first pass deletes columns that contain a zero (0) in BOTH the N and S+N rows.
2. The second pass through the matrix collapses columns with a zero (0) for N and a one
(1) for S+N; or one (1) for N and zero (0) for S+N. Columns are collapsed from right
to left except for the first column,which is collapsed from left to right.
Example:
Original categorical data matrix:
1
2
3
4
5
_____ _____ _____ _____ _____
|
|
|
|
|
|
| 1 | 1 | 1 | 0 | 0 |
|_____|_____|_____|_____|_____|
|
|
|
|
|
|
| 0 | 0 | 0 | 0 | 1 |
|_____|_____|_____|_____|_____|
After the first pass: Deleting columns
1
2
3
5
_____ _____ _____ _____
|
|
|
|
|
| 1 | 1 | 1 | 0 |
|_____|_____|_____|_____|
|
|
|
|
|
| 0 | 0 | 0 | 1 |
|_____|_____|_____|_____|
Second pass: Collapsing columns, first collapse
1
2
3
_____ _____ _____
|
|
|
|
| 1 | 1 | 1 |
|_____|_____|_____|
|
|
|
|
| 0 | 0 | 1 |
|_____|_____|_____|
DBM MRMC User's Guide (Windows OS)
Page 43 of 45
Second pass: Collapsing columns, second collapse
1
3
_____ _____
|
|
|
| 2 | 1 |
|_____|_____|
|
|
|
| 0 | 1 |
|_____|_____|
This is the final form of the matrix.
SPECIAL NOTE:
All of the recategorization options operate only on the data in program memory. The
original data file remains as it was. The current version of the software does not permit
you to save the recategorized data. This feature may be added to a future release.
DBM MRMC User's Guide (Windows OS)
Page 44 of 45
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Pepe MS. The statistical evaluation of medical tests for classification and prediction.
Oxford ; New York: Oxford University Press, 2004.
Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis.
Generalization to the population of readers and patients with the jackknife method. Invest
Radiol 1992; 27:723-731.
Hillis SL, Obuchowski NA, Schartz KM, Berbaum KS. A comparison of the DorfmanBerbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic
(ROC) data. Stat Med 2005; 24:1579-1607.
Hillis SL, Berbaum KS. Monte Carlo validation of the Dorfman-Berbaum-Metz method
using normalized pseudovalues and less data-based model simplification. Academic
Radiology 2005; 12:1534-1541.
DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more
correlated receiver operating characteristic curves: a nonparametric approach. Biometrics
1988; 44:837-845.
Metz CE, Pan X. "Proper" Binormal ROC Curves: Theory and Maximum-Likelihood
Estimation. J Math Psychol 1999; 43:1-33.
Dorfman DD, Berbaum KS. A contaminated binormal model for ROC data: Part II. A
formal model. Acad Radiol 2000; 7:427-437.
Metz CE. Some practical issues of experimental design and data analysis in radiological
ROC studies. Invest Radiol 1989; 24:234-245.
Metz CE, Herman BA, Shen JH. Maximum likelihood estimation of receiver operating
characteristic (ROC) curves from continuously-distributed data. Stat Med 1998; 17:10331053.
Tukey JW. Bias and Confidence in Not-Quite Large Samples. Annals of Mathematical
Statistics 1958; 29:614-614.
DBM MRMC User's Guide (Windows OS)
Page 45 of 45