COUNTER Reports Validation Tool:
Requirements
Introduction
Both COUNTER and NISO provide implementers of COUNTER Reports and the SUSHI
protocol with detailed documentation and guidance on how to create compliant products;
however, even with this assistance the rate of interoperability and compliance problems are
still high. These issues cost publishers and vendors collectively 100s of thousands of dollars
per year in added support costs (handling service issues related to non-compliance),
development costs (re-working code to address bugs and compliance issues) and potentially
in loss of revenue due to the inability to deliver usage statistics in a timely manner. For
librarians, issues with COUNTER and SUSHI increase the cost to manage their e-resources
and may compromise the quality of their collections due to the lack of accurate or timely
usage data.
The goal of this project is to create a web-based tool that will test compliance of COUNTER
reports delivered in tabular format or delivered in XML via SUSHI. Reports will be uploaded
to the tool or a SUSHI harvest conducted through the tool and a series of test will be
conducted on the result. Errors found will be highlighted in a report to the tester. The tool
will be made freely available on the web for all to use, though self-registration will be
required to allow tracking of the tools use and improve error reporting.
The tool will enable publishers, vendors and usage consolidation product owners to test their
implementation of the SUSHI protocol and COUNTER reports during the development
process, ensuring errors are caught and corrected before release. The result will be more
efficient deployment and improved interoperability. This tool could also be used by the
COUNTER audit to validate compliance to the required format specified in the COUNTER
Code of Practice and the COUNTER SUSHI Implementation Profile, reducing the price of
the audit and increasing the compliance rates by those publishers and vendors that use it as
part of their testing.
Statements of Scope
The following statements are intended to describe the high-level scope of the testing tool
being built and are presented as a set of user stories (US).
Testing COUNTER Reports - tabular form
US1: As a librarian/publisher that has generated any standard COUNTER report in tabular
(e.g. Excel) form I want to verify that the COUNTER report is formatted correctly and the
data values it contains meets formatting requirements.
“Standard Reports” means:
 Reports that COUNTER requires a provider to offer to be compliant (not all providers
must provide all reports). See Table 1 in the COUNTER Code of Practice for the
official list of reports).
 In priority order these would be (up for discussion):

















Journal Report 1
Journal Report 5
Database Report 1
Book Report 1
Book Report 2
Platform Report 1
Journal Report 2
Database Report 2
Consortium Report 1
Consortium Report 2
Book Report 3
Journal Report 1 GOA
Multimedia Report 1
Consortium Report 3
Book Report 4
Book Report 5
Book Report 7
“Formatted correctly” means:
 Report layout is exactly as specified in the Code of Practice
 Labels and column headings exactly match the Code of Practice, this includes
spelling, spacing and capitalization.
 Metric type names, when included on the report, exactly match the Code of Practice
(spelling, special characters and spacing)
 The report does not contain extraneous rows, such as blank rows within the body of
the report or extra rows at the end with non-report data or blank rows at the top
before the headings.
 Unless specified otherwise in the Code of Practice, the report does not contain items
for which there was zero usage for the entire reporting period
 Usage values are zero or positive numbers and do not contain empty cells unless the
entire month column in blank signifying usage for that month is not available in the
system (e.g. when month is a future month as when in June of 2016 a user asks for
usage from January to December of 2016 -- June through December must all be
blank, including total rows blank to be able to discern zero usage from data that is not
yet available.)
 ISxNs are valid or omitted if not available and do not contain values such as “N/A”,
“not specified” or “unknown”
 DOIs are omitted if not available and do not contain values such as “N/A”, “not
specified” or “unknown”
 Proprietary identifiers are omitted if not available and do not contain values such as
“N/A”, “not specified” or “unknown”
 Total rows and columns, when provided, add up correctly
 In Journal Report 5, years of publication back to 2000 are represented as individual
entries and the correct formatting for YOP labels used.
US2:As a librarian/publisher, if errors are found when validating my report I want to know
what the errors are and where in the report they were found so that I may efficiently report
them to others or correct them.
Testing SUSHI Harvesting of COUNTER Reports
US3: As a librarian/publisher I want to be able to verify that a content provider’s SUSHI
server (for which I have valid credentials) is operating correctly and returning properly
formatted SUSHI responses and COUNTER reports for any applicable standard reports
available from that content provider.
“Operating correctly” means:
 The SUSHI Server doesn’t require non-standard authentication
 Server accepts valid request parameters
 COUNTER report that matches the request or proper error is returned
 The Server responds to error conditions with the correct error codes and messages
(automated tests the simulate errors conditions may be required).
“Properly formatted SUSHI Responses” means:
 Responses can be validated against the COUNTER_SUSHI Schema
 Error codes returned are compliant with those listed in the SUSHI Standard
 Responses adhere to the expectations described in the COUNTER SUSHI
Implementation Profile
“Properly formatted … COUNTER reports” means:
 The COUNTER XML can be validated against the release 4 versions of the
COUNTER XML schema and COUNTERElements XML schema. The testing tool
should be prepared to handle multiple COUNTER Releases which requires for
example different schemata for validation.
 Responses adhere to the expectations described in the COUNTER SUSHI
Implementation Profile (some examples follow).
 Identifiers are included only when there are valid values
 Standard identifiers, like ISSNs and ISBNs are formatted correctly
 Datatypes, Categories and Metric Types are appropriate for the report and used in
valid combinations
 COUNTER XML does NOT include multiple-month totals or report totalsl (each
“Period” element represents exactly one month)
 For Journal Report 5,
o PubYr attributes represent single years of publication in “yyyy” format, and
all years back to 2000 are included if the ReportItem had usage for any
content published that year
o PubYr and PubYrFrom & PubYrTo are mutually exclusive
o PubYrFrom is less that PubYrTo and both are years in “yyyy” format.
o isArchive attribute is present and set to “True” when year(s) of publication
represent content items in an archive. IsArchive attribute may be omitted if its
value would be “False”.
 Unless explicitly required by the CoP, zero-usage items are excluded
“Standard reports” means:
 Reports that COUNTER requires a provider to offer to be compliant (not all providers
must provide all reports). See Table 1 in the COUNTER Code of Practice for the
official list of reports).
 In priority order these would be (up for discussion):
 Journal Report 1
 Journal Report 5
 Database Report 1
 Book Report 1
 Book Report 2
 Platform Report 1
 Journal Report 2
 Database Report 2
 Consortium Report 1








Consortium Report 2
Book Report 3
Journal Report 1 GOA
Multimedia Report 1
Consortium Report 3
Book Report 4
Book Report 5
Book Report 7
US4: As a librarian/publisher I want to be able to detect which reports are supported by that
content provider’s SUSHI server (for which I have valid credentials).
“Detect which reports” means that the testing tool would use the list of compliant reports
from the COUNTER vendor registry for the Platform and run test requests for each to see if
they are supported. Each report would be validated for compliance and a response of report
not supported (even if a compliant error) would be considered an error for this test since the
report is expected to be supported..
US5: As a librarian/publisher I want to be able to verify that a content provider’s SUSHI
server (for which I have valid credentials) responds correctly to error conditions.
US6:As a librarian/publisher, if errors are found when validating my SUSHI request, I want to
know what the errors are and where in the report they were found so that I may efficiently
report them to others or correct them.
“What the errors are” means exact element and problems are enumerated and where
possible expectations provided (e.g. Element <123> found X expected Y)
“Where in the report” means a meaningful position in the report with error is represented.
(e.g. (e.g. Line 72, offset 1: element <123> found X expected Y)
“Know… and efficiently report them” means user should be able to review errors on-line and
export or email or otherwise communicate those errors to others without having to retype.
US7 (optional): As a software developer I want to have access to open source SUSHI Client
software to provide a guide and starting point for building my own client application.
“Open source” implies a project on github or similar where client code is available
“Starting point for building” implies that the SUSHI client portion of the testing tool is
somewhat self-contained and can be used in other contexts.
Reporting Errors
The testing tool should capture and record the errors encountered during testing such that
they can be viewed on screen or output as a report. Errors should be clearly articulated so
that the reader can determine exactly what is wrong and where within the submitted file or
SUSHI response that the error occurred. If possible it would be nice to highlight the errors in
the context of the report or SUSHI response. (see US2 and US4).
Logging Errors
The testing tool will be used in two modes:
 developers will use the tool to validate their work as they develop new COUNTER
report or SUSHI implementations; and,
 auditors, librarians and content providers will test production version of reports and
SUSHI implementations to check for compliance.
The system should allow for logging of errors encountered; however, it is important to
distinguish between developer-run-tests and tests run on production versions of COUNTER
reports and SUSHI. The former will be expected to fail frequently and the latter not.
US8: As a user of the COUNTER Report Validation Tool I want the system to record if I am
testing a production version of a COUNTER report or SUSHI or a version that is indevelopment.
US9: As an administrator of the COUNTER Report Validation Tool, I want to be able to
forward errors found in production systems to the content provider concerned and include
details of the test, errors found and the contact information of the person conducting the test.
US9a: As an administrator of the COUNTER Report Validation Tool, I do not want to
forward errors to the content provider concerned if the tests were being conducted on a test
instance of the report/system, or if in test system or, if the tests were conducted by the staff
of the content provider.
US10: As an administrator of the COUNTER Report ValidationTool I want to conduct a
statistical analysis of tests run so that I can produce statistics by Platform, COUNTER
Report, SUSHI vs Spreadsheet, Test version vs Production version, Success vs Failure;
counting sessions, tests run, total errors encountered.
Architectural Consideration
The COUNTER Report Validation Tool should be accessible freely on the web, however, it
should be possible to track who is using the tool. Furthermore, the Testing Tool will be
accessible from COUNTER’s new website (a WordPress based site) and from USUS (a
WordPress based site.)
The new COUNTER website will also include a Registry of Compliant Vendors and
Publishers (The Registry) which will store information about compliant platforms, reports
they support, admin URLs, SUSHI Server URLs, configuration notes, contact details. The
Testing Tool should interface with the Registry to simplify data entry for users running tests
and to help accumulate statistics for tests run and test results at the Platform/Vendor level.
Running Tests
Before they are permitted to run tests, the user must identify themselves by providing the
following:
 Email (must be valid)
 Name
 Affiliation
 Role (Librarian, Auditor, Developer, Content Provider)
 Test mode (Production System/Report, Test Version of System/Report
US11: As an administrator of the Testing Tool, want to track who is using it by requiring
users to register with their email (username), name, affiliation, type of organization (library,
vendor, auditor), normal role for test (librarians, auditor, developer, etc.) and self-assigned
password. I further need the registration and recover of lost usernames and passwords to be
completely self-service.
US11a: As an administrator of the Testing Tool, I want to require registering customers to
complete a Captcha or similar to reduce the possibility of The Tool from being the target of
spam or robotic use.
Once information has been provided, the user will have the option to upload a COUNTER
Report, or conduct a SUSHI test.
COUNTER Report Upload
The system will prompt the user to identify:
What
Why
Required?
User ID/password
User must identify themselves for
the test. If they don’t have a login
they can register.
Yes
Role for the test (one of
“auditor, developer, librarian,
other)
Helps with logging and reporting.
Yes
Test mode (one of test-version,
production-version)
Helps inform logging and reporting.
Yes
COUNTER Release Version
(optional)
If multiple COUNTER releases are
supported by the tool, the release
version is required.
(first release of tool
will be Release 4)
Report (select from list of
supported COUNTER reports)
Controls the flow of the testing
No (if no entry, autodetect from the file
uploaded.)
Platform (option to search
registered platforms, type
platform name or leave blank
Ties the test to a given platform
and allows confirmation that the
Platform in the file is correct.
No (if no entry, autodetect from the file
uploaded.)
Browse for file to test
Select file on the user’s system
and upload for test
Yes
The user will then be given a file dialog to upload the file. Excel and TSV are permitted
formats.
US12: As a user of the Testing Tool, I want to be able to identify the COUNTER Report I am
planning to test then upload the report in Excel or TSV format.
Tests will be conducted based on the report requested. See Appendix A.
US13: As a user of the Testing Tool, I want appropriate tests to be conducted on the file I
uploaded so that I can see if the file complies.
US14: As a user of the Testing Tool, I want to be able to access the results of the test on
screen or on download so that I may evaluate the results.
US15: As a user of the Testing Tool, I want to have the option to forward the test results to
the contact listed for the vendor or to an email address that I enter (multiple addresses
supported) and have the uploaded file included in the email as an attachment.
.
SUSHI Client
The system will allow the user to select a Platform to test by using Registry, or type their own
Platform details if not. Data elements needed for the SUSHI Client to work include,
 Platform Name (Lookup/search Registry -- and auto-populate subsequent fields)
 Read-only notes on configuration
 SUSHI Server URL*
 Requestor ID*
 CustomerReference ID*
 Report* (select from list of valid reports for selected Platform as indicated in The
Registry. Allow entry of “Invalid Report”)
 Date Range* as From month-year and To month-year (default to 2 months ago)
When required data (*) is supplied, allow the user to submit the request.
US16: As a user of the COUNTER Report Validation Tool to test a SUSHI implementation, I
want to be able to leverage the SUSHI Server Registry to automatically populate the
Platform Name, URL, notes on configuration, reports available so that my testing efforts are
more efficient by just having to add or update data that is missing (I can overwrite any data
auto-populated).
US16a: As a user of the COUNTER Report Validation Tool, I want to be able to manually
enter SUSHI credentials in the event that Platform is not listed within the SUSHI Server
Registry.
Testing of SUSHI Responses and Error Conditions will vary by report. See US3 for more
details on test expectations (driven by COUNTER SUSHI Implementation Profile) and
section on Reporting Errors and Logging Errors.)
Efficient Modular Design
US17: As a software developer that may be responsible for maintaining or expanding the
COUNTER Report Validation Tool, I expect the code to be well organized and modular so
that I may quickly diagnose issues, add new test cases or expand existing ones.
US18: As a software developer that may be responsible for maintaining or expanding the
COUNTER Report Validation Tool, I expect the code to include methods to automate
regression testing so that I can be assured my changes did not negatively affect other areas
of the code
US19: As a software developer using the COUNTER Report Validation Tool I would expect
the tool to provide access to underlying features as a set of microservices (provided I have a
valid APIKey/credentials) so that I may directly interface with the testing framework used for
my application.
User and System Documentation
US20: As a user of the application I want to have documentation available to assist me in to
use the tool effectively.
US21: As a software developer I want access to comprehensive, easy to use documentation
about the application, its architectural design, how it is structured, its processing business
rules, underlying schemas, underlying web services, error handling, etc. so that I may
effectively maintain and extend the application as needed.
Appendix A: Sample Test Cases for
COUNTER Reports in Tabular form
Book Report 1
Test Name
Expected Result
Check cell A1 for proper report name (case
sensitive)
Book Report 1 (R4)
Check Cell B1 for proper report title (case
insensitive)
Number of Successful Title Requests
by Month and Title
Check Cell A4 for period covered label (case
insensitive)
Period covered by Report:
Check cell A5 for correctly formatted period covered
A5 contains period covered in the
form of 'yyyy-mm-dd to yyyy-mm-dd'
Check cell A6 for date run label (case insensitive)
Date run:
Check cell A7 to make sure it is a date.
Cell A7 contains rundate in the form
'yyyy-mm-dd'
Check cell A8 for the correct label
(cell should be empty)
Check cell B8 for the correct label
Publisher
Check cell C8 for the correct label
Platform
Check cell D8 for the correct label
Book DOI
Check cell E8 for the correct label
Proprietary Identifier
Check cell F8 for the correct label
ISBN
Check cell G8 for the correct label
ISSN
Check cell H8 for the correct label
Reporting Period Total
Check for correct date headings in cells I8:T8
First date column = first date in Period
covered in A7
Check for correct date headings in cells I8:T8
Last date column = last date in Period
covered in A7
Check for correct date headings in cells I8:T8
Make sure all headings in the range
are dates
Check cell A9 has correct entry for Total for all Titles
(case insensitive)
Total for all Titles
Check that totals in cells H9:T9 add up.
Verify that totals are correct
Check that cells A10:A213 all have a title -- blank
rows are not allowed.
All rows must have a title
Check the cells C10:C213 have a platform name blank rows not allowed.
All rows must contain a platform name
Check for valid ISBNs in cells F9:F213
ISBNs are ISBN-13, have dashes and
valid check-digits
Check for valid ISSNs in G9:G213
ISSNs, if present, have dashes and
valid check-digits.
Check for blank count values in cells H10:H213.
Blanks not allowed unless the entire column is blank.
All count cells must be non-blank
Book Report 2
Test Name
Expected Result
Check cell A1 for proper report name (case sensitive)
Book Report 2 (R4)
Check Cell B1 for proper report title (case insensitive)
Number of Successful Section
Requests by Month and Title
Check Cell B2 for section type label (case
insensitive)
Section Type:
Check Cell B3 for section type value
A non-empty cell
Check Cell A4 for period covered label (case
insensitive)
Period covered by Report:
Check cell A5 for correctly formatted period covered
A5 contains period covered in the
form of 'yyyy-mm-dd to yyyy-mm-dd'
Check cell A6 for date run label (case insensitive)
Date run:
Check cell A7 to make sure it is a date.
Cell A7 contains rundate in the form
'yyyy-mm-dd'
Check cell A8 for the correct label
(cell should be empty)
Check cell B8 for the correct label
Publisher
Check cell C8 for the correct label
Platform
Check cell D8 for the correct label
Book DOI
Check cell E8 for the correct label
Proprietary Identifier
Check cell F8 for the correct label
ISBN
Check cell G8 for the correct label
ISSN
Check cell H8 for the correct label
Reporting Period Total
Check for correct date headings in cells I8:T8
First date column = first date in Period
covered in A7
Check for correct date headings in cells I8:T8
Last date column = last date in Period
covered in A7
Check for correct date headings in cells I8:T8
Make sure all headings in the range
are dates
Check cell A9 has correct entry for Total for all Titles
(case insensitive)
Total for all Titles
Check that totals in cells H9:T9 add up.
Verify that totals are correct
Check that cells A10:A1250 all have a title -- blank
rows are not allowed.
All rows must have a title
Check the cells C10:C1250 have a platform name blank rows not allowed.
All rows must contain a platform
name
Check for valid ISBNs in cells F9:F1250
ISBNs are ISBN-13, have dashes and
valid check-digits
Check for valid ISSNs in G9:G1250
ISSNs, if present, have dashes and
valid check-digits.
Check for blank count values in cells H10:H1250.
Blanks not allowed unless the entire column is blank.
All count cells must be non-blank
Book Report 3
Test Name
Expected Result
Check cell A1 for proper report name (case
sensitive)
Book Report 3 (R4)
Check Cell B1 for proper report title (case
insensitive)
Access Denied to Content Items by
Month, Title and Category
Check Cell A4 for period covered label (case
Period covered by Report:
insensitive)
Check cell A5 for correctly formatted period covered
A5 contains period covered in the form
of 'yyyy-mm-dd to yyyy-mm-dd'
Check cell A6 for date run label (case insensitive)
Date run:
Check cell A7 to make sure it is a date.
Cell A7 contains run date in the form
'yyyy-mm-dd'
Check cell A8 for the correct label
(cell should be empty)
Check cell B8 for the correct label
Publisher
Check cell C8 for the correct label
Platform
Check cell D8 for the correct label
Book DOI
Check cell E8 for the correct label
Proprietary Identifier
Check cell F8 for the correct label
ISBN
Check cell G8 for the correct label
ISSN
Check cell H8 for the correct label
Access Denied Category
Check cell I8 for the correct label
Reporting Period Total
Check for correct date headings in cells J8:U8
First date column = first date in Period
covered in A7
Check for correct date headings in cells J8:U8
Last date column = last date in Period
covered in A7
Check for correct date headings in cells J8:U8
Make sure all headings in the range
are dates
Check cell A9 has correct entry for Total for all Titles
(case insensitive)
Total for all Titles
Check that cells A10:A50 all have a title -- blank
rows are not allowed.
All rows must have a title
Check the cells C11:C50 have a platform name blank rows not allowed.
All rows must contain a platform name
Check for valid ISBNs in cells F9:F50
ISBNs are ISBN-13, have dashes and
valid check-digits
Check for valid ISSNs in G9:G50
ISSNs, if present, have dashes and
valid check-digits.
Check for valid access denied types in cells H9:H50
All access denied types are valid
Check for blank count values in cells I10:I50. Blanks
not allowed unless the entire column is blank.
All count cells must be non-blank
Journal Report 1
Test
Expected Result
Nature
of failure
Cell A1 value
Exactly equals: “Journal Report 1 (R4)”
Failure
Cell B1 value
Case-insensitive match to: “Number of
Successful Full-Text Article Requests by Month
and Journal”
Failure
Cell A4 value
Case-insensitive match to: “Period covered by
Report:”
Failure
Cell A5 value
Contains data range in the form of: “yyyy-mm-dd
to yyyy-mm-dd”
From-date’s day is “01” and to-date’s day is last
day of the month.
Warning
Cell A6 value
Case-insensitive match to: “Date run:”
Failue
Cell A7 value
contains a date in the form of “yyyy-mm-dd”
Warning
Cell A8 value
Exactly equals: “Journal”
Failure
Cell B8 value
Exactly equals: “Publisher”
Failure
Cell C8 value
Exactly equals: “Platform”
Failure
Cell D8 value
Exactly equals: “Journal DOI”
Failure
Cell E8 value
Exactly equals: “Proprietary Identifier” without
any newline character between the words.
Failure
Cell F8 value
Exactly equals: “Print ISSN”
Failure
Cell G8 value
Exactly equals: “Online ISSN”
Failure
Cell H8 value
Exactly equals: “Reporting Period TotaI”
Failure
Cell I8 value
Exactly equals: “Reporting Period HTML”
Failure
Cell J8 value
Exactly equals: “Reporting Period PDF”
Failure
Cells K8:<LastColumn>8
Contain headings in the format of “mmm-yyyy”
Representing consecutive months. If date
formatted by Excel as a “Date”, all dates must be
the first day of month represented.
Failure
Cell A9
Exactly equals: “Total for all journals”
Failure
Cell B9
Is empty
Failure
Cell C9
Contains name of Platform being processed
Failure
Cells D9:<LastColumn>9
Equal sum of all values in column beneath (e.g.
D9=SUM(D10:D<lastRow)).
Failure
Except if total for the month is blank, then the
entire row must be blank. Only ending columns
can be blank (e.g. the user requested months not
yet processed and those months are represented
on the report -- they cannot have values.
Cells C10:C<last row of
spreadsheet>
Exactly equal name of Platform in C9 with no
empty cells.
Failure
Cell F10:F<lastRow>
Are empty OR contain valid ISSN formatted with
as nnnn-nnn?. Valid ISSN means final character
passes check-digit test.
Failure
Cell G10:G<lastRow>
Are empty OR contain valid ISSN formatted with
as nnnn-nnn?. Valid ISSN means final character
passes check-digit test.
Failure
Cells
H10:<lastcolumn><lastRow>
Check column by column. If the column total
(row 9) is blankContains zero or positive integer
value and not an empty cell, except in case
where the column total is blank
Failure
Cells A10:A<lastRow>
Are not empty. Also test for empty rows at the
end that may contain extraneous data -- if found,
fail the report.
Failure
References:
COUNTER Code of Practice, COUNTER Website, URL:
http://www.projectcounter.org/r4/COPR4.pdf
COUNTER-SUSHI Implementation Profile, NISO SUSHI Website, URL:
http://www.niso.org/publications/rp/rp-14-2014/
SUSHI Standard, NISO SUSHI Website, URL:
http://www.niso.org/standards/z39-93-2014/