5th Annual GILS Conference, April 7-10, 2003 Lisle, IL
Barriers to Interoperability
Technical and Not So Technical
William E. Moen
<[email protected]>
School of Library and Information Sciences
Texas Center for Digital Knowledge
University of North Texas
Denton, TX 72603
Interoperability: Volume or value
“…information production is growing at about 50%
a year…
Yet the amount of time people spend consuming is
growing only 1.7% a year…
As digital production and storage grow ever
cheaper…a critical task ahead will be to stop
volume from simply overwhelming value.”
John Seely Brown and Paul Duguid,
The Social Life of Information, 2002
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
2
Defining Interoperability
System-oriented definition

The ability of two or more systems or components to
exchange information and use the exchanged
information without special effort on either system
User-oriented definition
User’s ability to successfully search and retrieve, or
otherwise interact with information in a meaningful way
and have confidence in the results
 The condition achieved when two or more technical
systems can exchange information directly in a way that
is satisfactory to users of the systems (AAP)

Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
3
Assessing Interoperability

Binary
Interoperable
 Not interoperable


Continuum
More or less interoperable
 Acceptable levels of interoperability

Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
4
Fundamental assumptions
Your users will interact with resources other than
those available locally
 Your system will interact with other systems’
content
 Other systems will interact with your content
 Systems and content were often meant for local or
proprietary use

In the networked environment
diverse systems and varied content
must interact to serve users
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
5
The foundation: Communication protocols
Interoperability is about communication – albeit
between machines/software
 Two systems interchange bits/bytes
 Protocols (computer-to-computer communications)
govern the exchange of bits/bytes


Moen
Rules governing the structure, order, and content of the
bits/bytes/messages between machines
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
6
Protocols and record formats/markup
Protocols enable communication between systems
 Record formats/syntax/markup address resources
 HTML = HyperText Markup Language

Markup language for simple web documents
 Display oriented


MARC = Machine Readable Cataloging


Communications format for exchange of bibliographic
information
XML = Extensible Markup Language
Markup language for web environment
 Content oriented

Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
7
Factors affecting interoperability

Multiple protocols


Multiple data formats, syntax, metadata schemes


MARC 21, UNIMARC, XML, / ISBD/AACR2-based, Dublin Core
Multiple vocabularies, ontologies, disciplines

LCSH, MESH, AAT

Multiple languages, Multiple character sets

Multiple and disparate systems


Moen
Z39.50, HTTP, SOAP, etc.
Information retrieval systems, search functionality, etc.
Indexing, word normalization, and word extraction policies
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
8
Assumptions

Increasing interoperability factors

Increased complexity/costs to achieve
interoperability

How to identify barriers to interoperability:
In general?
 In specific applications?

Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
9
Interoperability Testing
Realizing the Vision of Networked Access to Library
Resources: An Applied Research and Demonstration
Project to Establish and Operate a Z39.50
Interoperability Testbed


A U.S. Federal Institute of Museum and Library Services
National Leadership Grant
Research and demonstration project to:




Establish a rigorous and trusted Z39.50 testbed
Develop methodology and metrics for assessment
From demonstration  operational testbed
Goal: Improve Z39.50 semantic interoperability among
libraries for information access and resource sharing
FOR MORE INFORMATION, VISIT THE PROJECT WEBSITE…
http://www.unt.edu/zinterop/
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
10
Components of the Testbed

Test dataset


Z39.50 reference implementations


Searches with known result records from dataset
Benchmarks

Moen
Z-client, Z-server, information retrieval system
Test scenarios & searches


400,000 MARC 21 records from OCLC
Results of test searches against reference
implementations
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
11
Interoperability findings

Successes:
Z39.50 profiles improve interoperability
 Z-clients and Z-servers can be configured to improve
 Common indexing policies support better interoperability


New problems:
Word extraction (what constitutes a word)
 Data normalization (special characters, diacritics)
 Indexing (leading articles, “first in field” searches)


Moen
What is good enough interoperability?
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
12
ZLOT project

The Library of Texas

A statewide virtual library to extend access to information
• Collection Boundaries
• Organizational Boundaries
• Format Boundaries

Z Implementation Component of the Library of Texas



Developing a standards-based approach for the virtual library
Identifying requirements for resource discovery across library
catalogs and online databases
Demonstrating feasibility and utility of resource discovery tools
FOR MORE INFORMATION, VISIT THE PROJECT WEBSITE…
http://www.unt.edu/zlot/
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
13
One approach for framing interoperability
Information communities / communities of practice
 Networked information retrieval occurs within and
across information communities
 Information communities

Focal community
 Extended community
 Extra community


Costs to achieve interoperability vary depending…
Within focal community
 Within extended community
 Between disparate communities

Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
14
Focal Community


Community agreements exist (e.g., standards and rules)
Interoperability factors reduced…more easily achieved
Libraries as
Focal Community







Moen
Relatively homogeneous
Standards-based MARC records
Content and structure prescribed by AACR
Commonly available access points
Use of controlled vocabularies
Z39.50 widely implemented
But…
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
15
Information exchange within/between…
Extended Community
(e.g., Cultural Heritage)
Extra Community
Focal Community
(e.g., Geospatial )
Focal Community
(e.g., Libraries)
Focal Community
(e.g., Archives)
Extended Community
Focal Community
(e.g., Museum)
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
Focal Community
(e.g., Geospatial )
Focal Community
(e.g., Natural History
Museums)
16
GILS as information community
Conceptualizing GILS as information community
 Way to discuss barriers to interoperability:

Types of data and resources?
 Common resource description scheme
 Commonly used database semantics?
 Common record syntax (e.g., XML)
 Commonly understood access points/search concepts?
 Commonly used controlled vocabulary?
 Common protocols for information retrieval and
exchange?

Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
17
Services context for interoperability
Information communities provide setting
 Information communities offer services

Value-added services to end users
 Infrastructure services for providers


Mapping a service environment …
Needs
 Services
 Service interfaces
 Content


Moen
… to identify interoperability barriers/challenges
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
18
Characterizing the networked library
Virtual library components
 An environment for provision of

Services
 Resources


Not a digital library


Not all resources available are in digital/electronic form
An evolving product that responds to
Users’ needs
 Available resources
 Current and emerging technologies

Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
19
Service-centric networked library
Management
User
Groups
SERVICES
Technology
Resources
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
20
Networked library components

User groups



Local, onsite
Distributed, remote


Local
Collaborative
Remote
Technology


Services





Local applications
Distributed applications
Standards
Management


Local
Collaborative
Resources

Information
• Local
• Distributed

Moen
Human
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
21
User-driven services
User
Group
User
Group
User
Group
User
Group
LIBRARY SERVICES
Library
Resources
Moen
Library
Technology
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
22
Building services for the networked library
What are the requirements for the services?
 What are alternatives for deploying the services?
 What technologies are available to support the
services?
 What standards are available?
 What are the tradeoffs of standards and nonstandards approaches?

Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
23
Services for the networked library

Infrastructure/internal







Moen
Collection identification,
selection, and
acquisition/access
Collection building and
digitization
Collection organization and
preparation
Inter-organization access
Trust and authentication
Billing and payment
Preservation and archiving

Patron





Resource discovery service
Resource access service
Reference service
Instruction service
Patron account service
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
24
Virtual library services environment
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
25
Architecture
PROVIDER LAYER
Communication Protocols
SERVICES LAYER
Communication Protocols
USER LAYER
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
26
Provider layer
Provider Type Dependent on VL Service
3
Digital
Content
Librarian
Library
Supplier
2
Resource
Discovery
Service
Virtual
Reference
Service
Resource
Sharing
Service
Document
Delivery
Service
1
Moen
USER LAYER
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
27
Services layer
Resource
Sharing
Authorization
Financial
Patron
Account
Reference
Education
Library
Information
Personalization
Resource
Discovery
Standard Information Interchange Among Services
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
28
Resource discovery: Services view
Personalize
Browse
Select
Collection
Evaluate
Search
Acquire
Save
Common User Interface
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
29
Resource discovery: Architectural view
CONTENT
Z39.50
TexShare
TRAIL
Others
OTHER
Directory
SOAP
SERVICES LAYER
Resource Discovery
Authentication
HTTP
USER LAYER
Moen
VC
Library
SOAP
Home
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
Office
Mobile
30
Finding / exposing

Finding


Exposing


I can find only what’s been exposed
How content is exposed determines the how and if I can find it
Common agreements on exposing and finding



Protocols
Syntaxes
Semantics
• Database
• Language


Moen
Goal is to connect users to information
Strategy is interoperability on multiple levels (protocol,
syntax, semantics, organization, etc.)
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
31
Service interfaces framework
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
32
Resource discovery: Operational view
Content
Content
Standards-based
Search & Retrieval
Protocols
Content
Content
OAI
Repository
Gateway
Common
Interface
Content
Content
Proprietary or
Non-standard
Search & Retrieval
Protocols
Content
Content
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
33
Infrastructure standards -- NISO









Moen
ANSI/NISO Z39.50: Information Retrieval (Z39.50): Application
Service Definition and Protocol Specification
ISO 10160 & ISO 10161: Interlibrary Loan Application Service
Definition & Protocol Specification
ANSI/NISO Z39.83: Circulation Interchange Protocol (NCIP)
ANSI/NISO Z39.84: Syntax for the Digital Object Identifier - 2000
ANSI/NISO Z39.85: Dublin Core Metadata Element Set - 2001
ANSI/NISO Z39.88: The OpenURL Framework for Context-Sensitive
Services (2002)
ANSI/NISO Z39.89:The U.S. National Z39.50 Profile for Library
Applications (2002)
Networked Reference Services (standards development underway)
MetaSearch Initiative (may lead to new standards activities)
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
34
Standards supporting services

Each standard enables different services
Networked information retrieval (Z39.50)
 Automated interlibrary loan (ILL)
 Circulation applications (NCIP)
 Reference linking (OpenURL)
 Resource description (Dublin Core)


Moen
But they can work together to support robust and
interoperable information access and resource
sharing services
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
35
Web services
Building Blocks for Constructing Applications
Moen
Service A
Service B
Resource Discovery
Authorization
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
36
Web services: Finding a service
Service
Registry
Service
Requestor
Moen
Service
Provider
Service A
Service B
Resource Discovery
Authorization
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
37
Web services: Enabling technologies
UDDI: Universal Description, Discovery and Integration
WSDL: Web Service Description Framework
SOAP: Simple Object Access Protocol
XML: Extensible Markup Language
UDDI - WSDL
Service
Registry
UDDI - WSDL
Service
Requestor
WSDL
SOAP
Service
Provider
XML-Based Information Exchange
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
38
Web services
Service
Registry
UDDI - WSDL
Publish
UDDI - WSDL
Find
Resource
Discovery
Service
WSDL - SOAP
Service B
Invoke Service
Authorization
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
39
ZING – Z39.50 International Next Generation
Make intellectual/semantic content of Z39.50 more
broadly available
 Make Z39.50 more attractive by lowering barriers to
implementation

Use of XML – to represent and encode data
 Use of HTTP – for transport
 Use of SOAP – for interaction between client and server
based on Remote Procedure Call (RPC)


Several initiatives: ZOOM, ez39.50, ZeeRex, SRW/U
FOR MORE INFORMATION, VISIT THE PROJECT WEBSITE…
http://www.loc.gov/z3950/agency/zing/zing.html
Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
40
Search/Retrieve Web Service – Overview


Builds on Z39.50 and web technologies
Retains Z39.50 concepts:






Web technologies: XML, SOAP/RPC, HTTP
Combines several Z39.50 features into two “operation types”



Moen
Results sets
Abstract access points
Abstract record schemas
Explain & Diagnostics
Search/Retrieve
Explain
Provides a standards-based method for representing and
issuing queries, and retrieving records
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
41
Interoperability and the user
Technical interoperability as foundation
 Value-adding services respond to user tasks
 Usability of the service

Reliable
 Predictable
 Ease of use
 User interface
 …

Moen
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
42
Final thoughts
Interoperability is essential
 Interoperability is multi-faceted
 Interoperability is ultimately based on agreements

Formal standards
 Community agreements


Agreements result from
Social processes
 Willingness to collaborate
 Balance of local needs in the networked environment


Moen
It’s the “not so technical” we need to work on
5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL
43