Modal Unit-Response Rates and Strategy:
A Study of Historical Response Rate Trends by Mode of Collection for Multiple Surveys
Michael Zabelsky
U.S. Census Bureau, 4600 Silver Hill Rd., Suitland, MD, 20746
Disclaimer
This report is released to inform interested parties of research and to encourage discussion. Any views
expressed on statistical, methodological, technical, or operational issues are those of the author and not
necessarily those of the U.S. Census Bureau.
Abstract
In survey processing, often times research and operations are viewed as two completely separate
disciplines. This study intends to bridge one aspect of the research and operations gap, using historical,
modal, statistical data analysis and response rate data visualization to make recommendations that impact
future operations decisions and objectives.
Key words: Statistical data analysis, data visualization, operations decisions
1. Thesis
This report suggests that, to synchronize the introduction of web-collection instrument enhancements,
online data resources, and online data visualization tools to coincide with sample reselection cycles will
ultimately help maximize electronic response rates over time. Increases in electronic response are highly
correlated with increases in overall response for this paper’s featured survey, the Quarterly Services
Survey. Integrated web applications and sample selection redesigns both involve long, and often
expensive, development lifecycles. As a result, numerous strategic benefits could result by pairing the
development and release of both of these critical activities.
In addition, this paper will explore how statistical agencies can use historical, modal, operations statistics
to determine future direct-mail package contents and establish multi-mode, first-contact operations based
on evolving reporting preferences. Ultimately, it is expected that when a survey design adapts to
respondent’s preferences or needs, an increase in first-contact response rates can occur. This increase will
also directly lead to decreases in the cost and size of follow-up operations and a reduction in overall
response burden. If measured consistently across different surveys, trends in modal response can be used
as a method of evaluation for making program comparisons and reconciliations. Tracking unit-level
completion rates by operation within one statistical period can give managers a measure of the
contribution of maintaining each mode of collection, and of the effectiveness of individual initial and
follow-up efforts. More accurately forecasting response trends, and therefore job sizes of cost drivers,
could also result in reduced processing center shop floor staffing and cost estimates.
2. External Environment – Risk is an Opportunity
The United States Postal Service (USPS) is an integral part of collecting Census Bureau survey data.
Over the past three decades, the federal government and the information technology industry in America
have dramatically changed the way people communicate. Because of these societal, technological,
communication efficiencies, and further fueled by the recession of 2007-2009, the USPS is now
1
threatened by decreased mail volume. An overall decline in mail volume has forced the unit cost of
mailing to higher levels. That trend will likely continue as USPS reacts to a changing external
environment. The pressures forcing corporate rebranding and restructuring may eventually result in
overall service changes. Even with imminent reforms already in place or looming, the Government
Accountability Office (GAO) is projecting future losses for the United States Postal Service in the
hundreds of billions of dollars through 20201. Using this time horizon, the increasing probability of a
high-impact event affecting both institutions warrants the Census Bureau’s consideration.
3. Motivating the Research
At a strategic level, the Census Bureau must continue to monitor survey collection practices in the wake
of USPS uncertainty and with the advent of new technologies. As part of a long-term risk mitigation
strategy, the Census Bureau should continue to reposition itself by designing more progressive and
technologically savvy online data collection tools. By investing in new collection practices, the Census
Bureau can better position itself for future mail shocks or changes in respondents’ attitudes towards the
usage of the mail system. In addition, these new processes and technologies will be focused on helping
to:
Improve first contact response rates
Set tough internal standards for improving electronic response rates
Strengthen internal controls that direct investment in technology and staffing devoted to creating
public-facing web content
Make strategically-timed enhancements to online reporting instruments and data resources to increase
user satisfaction and overall online usage
Develop dynamic response strategies that recognize and leverage modal response histories to
influence the type of survey outreach materials designed and method of delivery
4. Measurements and Participation
This study identified all current, multi-mode, Census Bureau Economic Directorate surveys that have
maintained historical response records and were willing to participate. Modal response data were
collected at the closeout of each statistical period. In this study, the source of all electronic response rates
originates from one or a combination of Census Taker and Centurion. Census Taker and Centurion are
the primary Internet data collection software applications used by the Economic Directorate at the Census
Bureau. In addition, an internal survey of Census managers was conducted to identify changes in each
program that may have influenced modal response rates over time.
5. Reliability and Limitations of Data
Various factors affect response, including exogenous factors such as socio-political and economic
reasons, and endogenous factors such as response follow-up strategies and sample variation. This
analysis does not specifically investigate the effect that sample variation has on response rates. Interested
parties may refer to www.census.gov/econ to learn more about data limitations and reliability of survey
data for surveys included in this report. The analyses in this report focus on the option and level of
Internet reporting as it is associated with other modes of collection and overall response, but do not test
for cause and effect and how an Internet reporting option, by itself, affects response.
1
U.S. Postal Service: Financial Crisis Demands Aggressive Action: GAO-10-538T:
http://www.gao.gov/assets/130/124292.pdf
2
6. Results
Data was only collected from surveys after the Internet became a response option. Because of the
staggered implementation and availability of electronic reporting instruments to individual programs over
time, the number of statistical periods of collected data varies. The surveys specifically included in this
study cover three different data collection frequencies:
Annual
Annual Retail Trade Survey (ARTS)
Information and Communication Technology Survey (ICT)
Annual Wholesale Trade Survey (AWTS)
Annual Capital Expenditures Survey (ACES)
Services Annual Survey (SAS)
Quarterly
Quarterly Services Survey (QSS)
-FEATURED SURVEYQuarterly Plant Capacity Utilization Survey (QPC)
Quarterly Financial Report, Manufacturing / Mining / Wholesale (QFR Man/Min)
Quarterly Financial Report, Retail (QFR Retail)
Quarterly Births Survey (SQ-Class)
Monthly
Monthly Wholesale Trade Survey (MWTS)
Over the past eight years a significant phase shift in response preferences occurred, as shown in Figures 1
& 2:
Figure 1:
Figure 2:
Figures 1 & 2: Figure 1 depicts the Internet response rate when electronic reporting in Census Taker or
Centurion was debuted for each survey. Debuts in Census Taker and Centurion take place as early as
2004. Figure 2 shows the Internet response rate for these same surveys as of 2011. The two figures help
to depict the shift of electronic reporting in the Economic Directorate at the Census Bureau.
Respondents, representing a variety of economic industries, are steadily migrating to online survey
reporting. Increases have been observed across the board in every survey studied. In nine of eleven
economic surveys studied, web-collection has met or outpaced response rates for all other modes of
collection combined, as depicted in Figure 3. Some surveys saw this dramatic shift in as few as two
statistical periods since the introduction of Internet reporting as a response option.
3
Figure 3:
Figure 4:
Figures 3 & 4: The first four graphs (Figures 1-4) are tied together using a color scheme. Debut Census
Taker and Centurion electronic reporting response rates are visualized in light shading in Figures 1 & 3.
The darker shaded bars categorize present day Internet response percentages in Figure 2, and also display
Internet response rate gains since an online reporting instrument’s introduction, in Figures 3 & 4. The
average annual percent change line in Figure 4 helps quantify Internet response rate growth for making
survey comparisons, and can also give the reader a clear understanding of the length of time an electronic
reporting option has been available for each survey by comparing the average annual growth rate to total
growth since debut. Note: SQ-Class is a quarterly survey, but only has three statistical periods of
recorded Internet modal data. This is the reason the average annual increase line is higher than the total
percent change bar in Figure 4.
Through a more detailed analysis of modal response histories for individual surveys, even more exciting
is the finding that under the right circumstances, increases in online response rates can be highly
correlated with increases in overall response. No other mode of collection can boast a positive correlation
with increases in overall response. As such, when it is offered, the Internet has become the only mode of
collection with a promising upward growth potential.
The Quarterly Services Survey (QSS) was selected as the featured survey for a number of reasons. Of all
surveys studied, it has the oldest online data collection operation2, established in 2004. It has maintained
four modes of collection since survey inception: Mail, Internet, Fax, and Phone. Of the study surveys,
QSS has the most statistical periods of recorded modal data that includes electronic reporting. Because so
many modal data points have been recorded, meaningful regression analysis can be performed comparing
mode-to-mode unit response rates, and mode-to-total unit response rates to show changing relationships
in response over time. There is also continuity in the QSS response strategy over many statistical periods;
2
Census Taker or Centurion online collection instruments only
4
it is a perfect example of the “traditional approach” of multi-mode response strategies defined in section
7.1 of this report. In addition, QSS has tracked historical, modal response statistics before and after one
sample reselection and four sample coverage expansions, allowing for the impacts of these macroprogram changes to be measured against existing trends in modal response data.
All modal response rates at collection-closeout since 2004Q1 are displayed in Figure 5:
Figure 5: For historical, time-series representations of modal response rates for a single survey, data are
being displayed as a stacked bar chart, as modal percentages of the total response rate percentage as a
function of sample universe size. See section 6.2 for an explanation of the why there are two 2006Q3
statistical periods being graphed.
For context, the major program changes occur during the following statistical periods:
Classification
Sample 1, named BSR2K:
Statistical Period(s)
2004Q1 – 2006Q3
Sample Reselection:
2006Q3, two periods of data displayed
Sample 2, named BSR06:
2006Q3 – 2011Q1
Sample Coverage Expansions:
2009Q1, 2009Q2, 2009Q3, 2010Q2
6.1. QSS Sample 1; BSR2K, 2004Q1 -2006Q3
The most apparent trend measured during the first sample, BSR2K, is the conversion of a large
percentage of responses from the paper to the electronic mode. Figure 5 shows this electronic reporting
growth between quarters 2004Q1 -2006Q3. From the BSR2K sample’s origin to maturity, a 19%
increase in electronic response rates was measured, with an almost equal and opposite decrease in mailback forms being received. It appears that once a BSR2K sample respondent reported electronically for
the first time, not only did they continue to complete the survey, but they became conditioned to report
5
via the Internet. This established a low electronic reversion rate, observed as steady increases in the blue
Internet response rate bars in Figure 5. A low reversion rate laid the foundation for Internet response rate
growth to build on previous gains during almost every statistical period until the sample reselection in
2006Q3. While a very promising tendency, this trend indicates that the electronic instrument had not yet
met its full growth potential. At the start of the sample, a time-sensitive opportunity existed to increase
overall online usage.
Based on collection records, Census knows that 36% of all BSR2K firms were eventually willing to
report electronically. But Census must also note that there was unfulfilled growth potential based on two
measured factors: 1) the continuous increases in electronic response until a major program change that
introduced new, unconditioned respondents to the survey, combined with 2) a maximum electronic
response rate being achieved during the last statistical period of BSR2K, or at sample maturity. Could the
Census Bureau have done more to secure each electronic response sooner in the sample’s maturation? If
so, would the continual increases in electronic response have driven the BSR2K sample to finish at a
higher rate?
Even with the strong, negative correlation between mail and Internet response rates in BSR2K, it is during
this sample’s data collection that a positive correlation begins to appear between increases in electronic
response and increases in overall response, as shown in Figure 6. For QSS, there are two important
factors to consider when reviewing the linear regression for mode-to-mode or mode-to-total response
rates. Meaningful data analysis only takes place within each sample. When response data are plotted
across samples, no relationship appears to exist, as shown in Figure 7. A second, important factor that
contributes to higher overall response rates is record management within a sample by analytical staff that,
in effect, shrinks the sample size of cases being contacted in future statistical periods. More problem
cases are being removed from each subsequent statistical period’s collection efforts or imputed,
improving the quality of the average case being contacted in a sample through to a sample’s maturity.
Figure 6: Correlation = .81, R-squared = .66
Figure 7: Correlation = .01, R-squared = .1
6.2 QSS Sample Reselection, 2006Q3
The Quarterly Services Survey (QSS) underwent a sample reselection in statistical period 2006Q3, from
which a fraction of cases were dropped completely from the sample. A fraction of cases remained exactly
as they were in the sample or experienced a NAICS coverage change, and some corporate entities
remained with a change to individual firms being selected to participate. The sample is rounded out by
completely new firms being selected to participate. It is from the latter two categories that we find our
most noteworthy response event for the sample reselection during 2006Q3. Internet response rates are
immediately slashed by 10%. That decrease is felt in overall response rates too, which drop 11%. The
difference between the two 2006Q3 bars in Figure 5 is the substitution of BSR2K sample “drops” for
6
BSR06 sample “adds”. The decreases in both Internet and overall response rates are most likely
attributed to a respondent seeing a QSS form for the first time.
6.3 QSS Sample 2; BSR06 – Pre-expansion, 2006Q3 – 2008Q4
Prior to the coverage expansions that start in 2009Q1, the BSR06 sample sees electronic response
associated with new, overall response rate gains. It is during this period of time that the electronic
reporting instrument eclipses the joint contributions of all other modes of collection relative to overall
response. Unlike in BSR2K, a strong, negative relationship does not exist between electronic and mail
response during BSR06. Mail response can be described as a steady, moderate decline between 2006Q3 –
2008Q4, while electronic response mirrors large gains attained in BSR2K, as shown in Figure 5. As a
result of the absence of a strong, negative correlation between electronic and mail response during
BSR06, the gains in electronic reporting are now more highly correlated with increases in overall
response rates. This correlation is shown in Figure 8. Online data collection still does not appear to have
reached its maximum growth potential by 2008Q4. The systematic pattern of electronic response rate
gains over both samples proves that respondents were satisfied with reporting using the online instrument.
Figure 8: Correlation = .91, R-squared = .83
6.4 Tactical Planning Recommendations
The Census Bureau has an opportunity to maximize Internet response rates and improve overall response
rates when making sample modifications. Based on what was measured over the course of the BSR2K
sample and during the sample reselection, a time-sensitive opportunity exists to aggressively target new
respondents during a reselection period and promote the electronic reporting instrument. Eye-catching
promotional materials, and a “year 1” strategic communications plan are encouraged. The goal is to
design and deliver survey materials that compel a respondent to initiate the Internet reporting log-in
process because of the known, high retention rate of online responders.
A second planning recommendation states that, for surveys that experience five-year sample reselections,
all major online instrument redesigns should be timed to coincide with those reselection periods in order
to maximize the effect of the improvement. Survey designers should ensure marketable changes that
enhance the overall online experience. For the most part, new respondents making their first impression
of a survey are introduced in large groups during periods of sample reselection or coverage expansion.
The timing of reselection cycles are known far in advance, and are less likely to be delayed because of
budgetary constraints compared to proposed coverage expansions. Therefore, for planning purposes, it
may be easier to sync the timing of Internet reporting enhancements with reselection cycles, not coverage
expansions.
7
A third planning initiative recommends experimenting with the transition of broadening a respondent’s
online experience beyond that of just reporting for a survey; make the sharing of information mutually
beneficial. Statistical agencies spend considerable resources getting survey forms into an audience’s
hands to generate response, and even more resources to securely offer and promote an electronic mode of
collection as a response option. If agencies do not in turn highlight applicable statistical products already
being created as part of a respondent’s online experience, or even go so far as to integrate statistical
products into the electronic reporting experience itself, then a valuable opportunity to self-promote the
work of the organization to the general public during their visit to the agency’s websites is being lost.
6.5 Sample 2; BSR06 including Expansions, 2006Q3 - Present
The QSS response rate has fluctuated greatly during the survey’s five major program changes. After all
of the expansions, the QSS sample size more than tripled. The overall response rate of seventy five
percent is actually the same in the first and last period of collected response data. What has changed over
that period of time is the importance of the electronic reporting option that contributes to overall response.
In eight years it has become the preferred method of reporting, evolving from around twenty percent of all
responses to now contribute over sixty percent of all responses each statistical period, as seen in Figures 5
and 10.
Figure 9:
Figure 10:
Figures 9 & 10: Both figures display response rate changes from the baseline rates established at the
inception of the survey, in Statistical Period 2004Q1. Figure 9 shows period-to-period fluctuations in
overall response. Figure 10 shows period-to-period fluctuations in Internet response.
The resilience of the electronic reporting option is evident, displayed by how quickly the Internet
response rates rebound following expansion quarters, as seen in Figure 5. Once again, the final statistical
period of recorded modal data in 2011Q1 has Internet finishing at an all-time high for the survey,
indicating the electronic reporting instrument has most likely not reached its full growth potential.
6.6 Respondent Type, Feature Driven
Not all surveys conduct sample reselections every five years. The Quarterly Financial Report (QFR) is an
example of a survey that continually refreshes its sample. The survey form is completely different than
8
QSS; it has many more data elements and is more time consuming for respondents to complete.
However, a positive correlation still exists between increases in electronic response and increases in
overall response. High electronic reporting retention rates can be achieved by knowing a survey’s
audience and tailoring features of the reporting tool to fit their style and needs. Accountants have the
ability to print and file a proof of response certificate after successfully submitting a response
electronically. No other QFR collection option offers tangible evidence indicating a completed survey.
The importance of this type of evidence increases in line with a survey’s complexity and response burden.
Even though there are stark contrasts between reporting for the QSS and QFR surveys, by building an
online environment with desired features that reinforce the preferred mode of collection, a survey can
create and exploit opportunities to improve overall response rates.
7. Solution Vision and Scope
Modernizing multi-mode response strategies using paradata is critical to this paper’s thesis. Defining
existing and new response strategies is an initial step towards using historical, modal, operations statistics
to determine future direct-mail package contents and establish multi-mode, first-contact operations based
on evolving reporting preferences for a sample survey. The current Quarterly Services Survey (QSS)
response strategy is defined as an example of a typical, existing, multi-mode data collection approach.
An overview of the new 2012 Commodity Flow Survey (CFS) response strategy will be introduced as a
way to redefine a multi-mode response strategy that leverages response history and previous collection
operations. The new CFS response strategy can be applied to other sample surveys. Section 7.2 will
propose a new utility for electronic data collection and a new channel for data dissemination. Building on
section 7.2, an envisioned, future state of origin-destination surveys is being proposed in Section 7.3 as an
example of how to expand on electronic reporting objectives defined in a Census Bureau Director’s
ICES-III presentation. The four objectives from the Census Director’s ICES-III presentation appear at the
start of the next three subsections, 7.1, 7.2, and 7.3.
7.1 Two Types of Multi-Mode Response Strategies, Defined
Objective: facilitate and accelerate reporting3
7.1.1 Quarterly Service Survey (QSS), existing response strategy
Traditional Approach - a series of separate, single-mode operations conducted sequentially over the
course of one statistical period’s data collection, resulting in multi-mode data collection for the overall
statistical period. The traditional approach is typically only reactive based on direct respondent requests.
7.1.2 Commodity Flow Survey (CFS), new adaptive response strategy
Responsive Approach - multi-mode operations conducted simultaneously over the course of one statistical
period’s data collection, where multiple modes of collection are being used to satisfy the same first
contact or follow-up respondent contact dates. Language in outreach materials, tools and scripts are being
designed to specifically address each of these unique response scenarios and encourage electronic
response, and respondents are being qualified for different Census outreach operations based on past
response history and contact information collected in previous statistical periods.
7.1.3 Recommending the adaptive response strategy
This paper recommends more multi-mode Census Bureau surveys adopt the responsive approach.
Benefits of the traditional approach are in its simplicity and continuity. Stable, consistent response
environments have allowed for response rate measurements to become trends over time. Compared to the
3
Lee, Ronald; Mesenbourg, Thomas, The Electronic Reporting Option – Does It Make Sense or Cents?
ICES-III Presentation Slide 3
9
responsive approach, the traditional approach requires fewer total operations per statistical period.
However, though the responsive approach requires more total operations, the additional operations would
take place within the same work units, would represent smaller job sizes, and would be a natural way to
break-up outreach operation work for large surveys at Census’s processing center. Major costs associated
with planning, programming, and testing would still only be incurred once, increasing the value per
operation contributed by each work unit, while transitioning to more personalized outreach operations.
The responsive approach can still foster a stable reporting environment over time, just with more response
scenarios.
An advantage of implementing the responsive approach comes when a survey achieves high levels of
electronic response. The subsequent, letter-only, direct-mail operations targeted to past electronic
reporters in a sample cost less to print, are easier and faster to assemble, and represent lower and more
uniform package weights which reduces postage costs. In fact, for the Commodity Flow Survey (CFS),
almost all electronic reporters provided a fax number as part of a previous response. Faxing letter-only
outreach materials to generate electronic response is even cheaper than mailing them. For the 2012 CFS,
a complete response analysis comparing the effectiveness of form versus letter-only mail and fax
operations is not possible at this time. The survey is still conducting data collection. However, what has
been measured, is that while overall response rates have declined since the first quarter of data collection,
perhaps in part because of a significant reporting burden, the proportion of firms responding
electronically has increased over the course of conducting the survey.
7.2 Infusing Data Visualization through Electronic Reporting and Dissemination
Objective: transform to an E-Gov environment4
To broaden influence and attract new data users, combine:
Informing the public using
data visualizations
Online survey reporting
x
The creation and publishing of new data visualizations costs time, resources, and money. However,
statistical agencies can market these visualizations for free using existing touch points with respondents.
One way is to tap into web traffic flows from respondents reporting for surveys electronically by
showcasing relevant visualizations for each particular reporting group. When a respondent logs into the
Internet to report for a survey, statistical agencies can use metadata associated with a survey’s User IDs
and Passwords to drive custom web creation and personalize web content. Statistical agencies can further
improve data dissemination to respondents by grouping data visualizations at national, state, or local
levels to help create profiles or special reports that integrate:
Demography
Industry
(if applicable to survey)
Geography
Organization Scope
(if applicable to survey)
For instance, if the Census Bureau is asking questions for a manufacturing survey only, then upon
completion of electronic data collection, the user can be directed to industry, demographic, and
geographic data that pertains to the scope of what they do as an organization. A respondent can be
4
Lee, Ronald; Mesenbourg, Thomas, The Electronic Reporting Option – Does It Make Sense or Cents?
ICES-III Presentation Slide 3
10
converted into a data user by introducing statistical products catered to their specific industry, geography,
or other characteristics. Adding value at the point of reporting could compel future respondents to report
to Census surveys not only out of legal requirement or obligation to country, but also out of direct interest
in the work of an organization. Dissemination using the electronic reporting tool could add more context
to the reporting experience for a survey, and could be a natural bridge to show respondents how Census or
sponsor agencies are using their responses or how a survey applies to them. If executed effectively, and
high electronic response rates are achieved and maintained, this strategy paired with the responsive
outreach approach in 7.1 could translate into significant processing cost savings.
Implementing an integrated electronic reporting data visualization marketing strategy can act as a vehicle
to deliver innovative forms of statistical products to new users. However, within the scope of this
recommendation, the true catalyst to spark a cultural shift in the way an agency conducts business must
include infusing data visualization consistently through data collection, internal data review and analysis,
and data dissemination. The following survey concept is one example of how this can be achieved, and is
an immediate opportunity to expand the applicability of an existing, custom, mapping tool more broadly
through new functionality that affects different business processes.
7.3 An Origin-Destination Survey Concept
Objectives: provide value-added services to businesses; improve data quality and reduce processing
costs5
Currently, major Economic Directorate components and functions for individual surveys, such as
electronic reporting, internal data review and analysis, and data dissemination, are often completed in
separate systems. In the Economic Programs Area, one prospect to pilot an initiative where electronic
reporting, internal data review and analysis, and data dissemination are all components housed in the
same software application is the Commodity Flow Survey (CFS). The CFS paper and electronic data
collection tools ask respondents to report for total number of shipments originating from an establishment
location. In addition, the CFS collects a sample of these total shipments in the form of domestic and
foreign shipment characteristics by destination and mode of transport. Reporting burden can be
significant, approximately two hours per form, as respondents are instructed to select and document a
sample of up to 40 shipments. An electronic data collection instrument has recently been developed for
2012; this is the first time online reporting is being offered for the CFS. A challenge during development
was incorporating the two-page shipments table into a user-friendly, online format.
Prior to 2012, reporting for this section of the survey could only happen by respondents filling out the
two-page shipments table found within the paper form. Online reporting for the 2012 Commodity Flow
Survey gave respondents the option of either uploading a spreadsheet of data, or navigating 40 individual
shipment screens on the Internet reporting tool, Centurion. Both online options have already been tested
and proven to be effective in the field, and it is not being suggested they be replaced moving forward.
However, a new reporting option is suggested, that could give the respondent the opportunity to map their
shipments interactively while building the full shipments table off-screen. At log-in, metadata associated
with an establishment’s User ID and Password positions and zooms a “You Are Here” on a map. The
user then proceeds to enter different domestic and foreign shipment destinations, creating single and
multi-mode shipment paths. After each shipment path is locked-in, a pop-up box allows the user to fill out
remaining shipment characteristics before proceeding to draw the next shipment path. Respondents
would repeat for the full sample of shipments on one map, and then be prompted to review their answers
prior finalizing a submission.
5
Lee, Ronald; Mesenbourg, Thomas, The Electronic Reporting Option – Does It Make Sense or Cents?
ICES-III Presentation Slide 3
11
Developing an interactive electronic collection instrument could make online reporting for the survey
easier and more enjoyable. More experienced Internet users might find the design more user-friendly.
Many respondents may already be familiar with this type of interaction through using services like
Google Maps to get driving directions. If executed and marketed correctly, the instrument design could
potentially increase online response rates over the course of the four quarters of this quinquennial survey.
Real data quality benefits could be achieved by more effectively engaging each respondent, receiving
more pre-edited data, and allowing Census analysts to review web-maps of shipment data as they were
reported in addition to data tables from edits or queries, as a separate component of the same software
mapping application. Using a sidebar of all shipment characteristic variables, an analyst could select one
or a combination of variables to assist in data review during data collection. After variable selection of,
say, a shipment value range ($), mode of transport, and city, an animation could be run, skipping from
one ID shipment map to the next, in a string of short time increments.
The new reporting tool could also be used during internal data analysis. Data quality edits built into the
mapping tool serve as another way of isolating groups of records for analyst review. The review process
could simply take place within one firm’s responses to a survey, or it could span survey response for
groups of firms with similar characteristics, such as region or commodity classification. The review
process could also isolate all records failing the same edit within a given timeframe, identify outliers, or
find failures for any combination of inter-related survey variables tied together by edit checks. Paper data
capture records could be loaded to the tool to take advantage of the interactive review environment.
Individual shipment paths overlaid on one map could be another way to review a limited amount of
reported data at the same time. But as the volume of shipment records being visualized on one map
increases, the visualization feature utilized by the interactive tool would have to change. Heat maps, with
colored shades representing quantity ranges, would be a good way to visualize, say, total weights of all
reported domestic railroad shipments originating in Los Angeles by destination location during April
2012.
In this example, the geographical destination quantities would be displayed as color-shaded states, cities,
or counties where rail shipments originating from Los Angeles were eventually delivered. A month-tomonth animation of this query could be useful in displaying the established rail distribution channels
stemming from L.A. A small change to the query could show the pattern of individual commodity
groupings being transported over the course of a full year by rail, and give an analyst insight into types
and volumes of commodities being imported by sea to the port of Los Angeles6. Colored, shaded, heat
maps displaying data by selecting survey variables are the most natural bridge from shipment path micro
data review to statistical dissemination with the envisioned interactive tool.
Incorporating data visualization into the means by which micro- and macro-data is reviewed could have
quality implications, and by augmenting and enhancing the abilities of analysts during this step, one could
increase the speed of the review process, discovering new patterns in the reported, edited, and publication
data, and giving analysts a deeper understanding of the survey data. A robust, front-end, interactive,
mapping, electronic data collection instrument and review tool with the power of time-series animation
and variable series selection can also be used seamlessly for data dissemination.
With open architecture software development7 and a sound marketing strategy in place connecting the
Census data provider base, historical, sponsor agency dissemination products could be pulled in reverse to
6
“California’s Ports: The fickle Asian Container” The Economist: http://www.economist.com/node/21543551
“the confluence of business and technical practices yielding modular, interoperable systems that adhere to open
standards with published interfaces.”
7
12
service the very people providing the source of Census data on a just-in-time basis, using the same frame
of reference as the online reporting tool. Development costs could be shared across the agencies
sponsoring and administering the survey. So, instead of maintaining one tool for electronic reporting,
converting and transferring data to use a different system for editing, and transferring and repackaging
data again for dissemination, a coordinated effort could produce a multi-faceted interface with
functionality that spans multiple business processes. A singular focal point could eventually produce a
tool with quality features unmatched by components developed and maintained in isolation. The breadth
of this development work would span three or more directorates within the Census Bureau and at least
one sponsoring agency. There are much simpler ways to use the electronic reporting instrument as a
dissemination tool using static web-content at no additional cost to a sponsor. Static content can still
result in a personalized online experience using any variable in a Centurion input file. Whether the
content is static or dynamic, this could be one aspect of a broader, agency-wide dissemination strategy.
8. Conclusions
Statistical agencies must build the competency to directly connect with and hold the interest of all
respondents. At the same time, these agencies must also become better at promoting their statistical
products and increasing the reach of their dissemination efforts. These strategies are critical in the
uncertain future of government funding, respondent tendencies, and public need for vital data. The
individual strategies discussed in this report do not come without risk. For instance, disseminating poor
quality data visualizations using the electronic reporting instrument, or prematurely launching an
interactive tool being developed for public use could give respondents a negative impression of an
agency’s usage of their survey responses. These mistakes could negatively affect future response rates.
Furthermore, a greater dependency on Internet reporting and electronic data dissemination can lead to
increased IT security threats. Census data and corresponding visualizations would likely come under
increasing public scrutiny. Allowing respondents to comment in a public forum on Census visualizations
would require internal public relations oversight. In-house, Census might want to establish an expert
team of graphics designers and editors to coordinate the standards for and creation of the new visuals
meant for public consumption. Overcoming these obstacles may be justified, because as budgets for all
government agencies come under closer review by Congress, there is a growing need to prove the value
of Census data to the general public. But the greater motivation, and the responsibility that Census has as
a body of public servants, is to make statistics relevant and meaningful to the various survey groups who
support the mission of the Bureau with their time and responses.
Evolving social norms underscore the need for increased Internet reporting options. Statistical agencies
must continue to improve these response options.
Added benefits of online reporting that reinforce this progression are:
Quicker response and faster turn-around time to exclude cases from follow-up contact operations
Data entry is done by the respondents and not by Census Bureau staff
Built-in edits can help correct data that is entered incorrectly by respondents
Cleaner flow of questions and built-in skip patterns
Entered data can be stored more easily for future reference by respondents
After management has made the commitment to support an electronic reporting option as a mode of
collection for a survey, then consider the cost to develop the online instrument as money spent. The
future of generating processing cost savings for multi-mode surveys with an electronic reporting option
will happen jointly by: 1) identifying respondents’ preferred mode of reporting by referencing individual
response histories, and translating that preference into a categorical assignment of future first-contact
13
outreach operations, while 2) reducing the volume of mail-back, fax-back, telephone follow-up, or
personal visits by maximizing electronic response rates earlier in an observation’s lifespan in a sample
and survey.
By doing so, survey managers will begin to minimize the hand touching of paper forms, the additional
processing steps associated with routing, capturing, and storing paper records, the redundancy of outreach
efforts that result from not recognizing and targeting preferred collection modes, and the costly human
capital required to personally follow-up with individual respondents. Processing cost savings can be
monitored and reinvested into web-development during previously frozen periods of online content
design and funding, in preparation for dissemination that coincides with macro-program change periods
for each survey. The end goal is to not completely eliminate all modes of collection other than the
Internet, rather, to use the other modes to drive the maximum amount of traffic to the web for reporting,
and to spark public interest in the usefulness of statistics.
14
Acknowledgments
A special thanks to Nancy Piesto for giving me the authority to conduct the study, and for being
instrumental in collecting the modal response rate data from Census managers. Thank you to Michael
Kornbau, Michelle Karlsson, Michael Padgett, Ian Hull, and William Davie Jr. for reviewing the paper,
providing comments, helping to reorganize the materials, and assisting in the finalization of the report and
graphics for publication. A special thanks to Ian Hull for editing the paper, and to Michael Padgett for
producing the motion charts in the conference presentation.
References
“California’s Ports: The fickle Asian container.” The Economist. China and the paradox of prosperity.
2012, January 28th – February 3rd. Retrieved January 28, 2012, from
http://www.economist.com/node/21543551.
Guertin, Nick (2010). “Naval Open Architecture Contract Guidebook For Program Managers, Version
2.0, June 30, 2010, Washington, Navy Yard, D.C.: Department of the Navy: 2.
Herr, Phillip (2010). “GAO-10-538T, U.S. Postal Service: Financial Crisis Demands Aggressive
Action.” Testimony Before the Subcommittee on Financial Services and General Government,
Committee on Appropriations, U.S. Senate, March 18, 2010, Washington, D.C.: United States
Government Accountability Office: 1-9.
Lee, Ronald; Mesenbourg, Thomas (2007). “The Electronic Reporting Option – Does It Make Sense or
Cents?” Proceedings of the Third International Conference of Establishment Surveys, June 18-21, 2007,
Montreal, Canada: American Statistical Association: Presentation Slide 3.
15