ADS Dataset Verification and Resolution Services
This document contains information about the Dataset Verification and
Linking efforts underway among the
NASA Archives and Data Centers and the University
of Chicago Press (publisher of ApJ, AJ and PASP).
This activity has taken place
under the auspices and guidance of the
NASA Astrophysics Data Centers
Executive Council (ADEC)
and aims at fulfilling the promise
of further integrating astronomical literature and the on-line data
it is based upon.
The NASA Astrophysics Data System
(ADS) will provide the tools necessary to publishers and users
at large for both dataset verification and linking through
stable, top-level services that can be maintained for the foreseeable future.
Links created to datasets from on-line manuscripts will always refer to
a dataset via a URI created using a well-defined identifier, and the
URI will be turned into one or more URLs in real-time by a central
resolver provided by ADS.
This will provide a high level of reliability and persistence to the links,
as well as providing an upgrade path into any future VO efforts in this
Dataset citation, verification and linking will work as follows:
ADS will take the responsibility of maintaining services that are
aware of all relevant datacenters that may have datasets available
on-line, and which datasets are available from which data centers.
- Astronomy data centers and archives will start attaching
permanent dataset identifiers to the data they distribute.
- Astronomers will write papers referencing the dataset they have
used. As per the instructions given to them by the AAS, they will
start using the appropriate markup to identify datasets in the papers.
- During the publishing pipeline, UCP will extract the identifiers
and send a query to a central dataset identifier service (hosted by
ADS) to find out if (a) the dataset is valid and (b) a URL can be
associated to it.
- The central dataset identifier verification service will query a number of
(relevant) datacenters using its own protocol, will cache the results,
and will return a status flag to the UCP query, indicating if a
dataset is known or not.
- For the dataset identifiers that are known, URLs can be built
by using the base URL of a dataset identifier resolver and the dataset
identifier itself, e.g.
If the verification is successful, UCP should include such URL in its
- When the article goes on-line, a user clicking on the link
associated with the dataset will be taken initially to the URL above.
What happens next depends on whether ADS has one or more datacenters
claiming to have data relative to this dataset (there could even be
different mirror sites for a given data center). If only one final
URL is available for the dataset in question, our cgi script can
simply forward the user to it. If more than a single URL is
available, we could display a simple menu listing all the information
we have about the available links.
In order to allow easy integration of this effort in the emerging VO
framework, the ADEC has decided to adopt a syntax for the dataset
identifiers which is consistent with the current
Identifier Proposed Recommendation
(Plante et al 2003).
This adoption will facilitate integration of these identifiers and the
tools that manipulate them in the VO.
According to the IVOA Identifiers Draft, the general URI format
for an individual dataset identifier is a string of the kind:
While we refer the reader to the recommendation
for a full explanation of the syntax, a few things are worth pointing out:
- Use of the ivo:// scheme denotes the fact that the rest of the
identifier should be interpreted as a string abiding by the
IVOA Identifiers specification, and that
the identifier and the resource it refers to have been registered with
an IVOA-compliant registry.
- AuthorityId is a naming authority registered within the IVOA
community; the use of this string within the identifier
establishes a namespace within which the rest of the identifier
can be considered unique. In general, the AuthorityId does not need
to correspond to a specific institution but rather to an entity that has
been granted use of the namespace.
- ResourceKey is a name for a resource that is unique within the
namespace estabilished by the AuthorityID. In general it will
correspond to a unique resource made available to the VO by or on
behalf of the AuthorityID. A typical example of a ResourceKey
in this context is a data collection generated by a particular project
- PrivateId represents a unique string within the ResourceKey and it
denotes a particular dataset belonging to the collection.
Using Dataset Identifiers in the Literature
Given the fact that much of the VO infrastructure is still under
design and development, the ADEC has decided on a specific recommendation
for referring to dataset identifiers in the astronomical literature.
The general form of these identifiers is:
Comparing these identifiers with the general IVOA syntax we can make
the following observations:
- No protocol scheme has been specified. This is due to the fact
that until IVOA-compliant registries are available, and AuthorityIds
can be established by them, it would be incorrect to claim that these
identifiers are in fact IVOA compliant.
However, it is to be expected that these identifiers can be resolved
as IVOA identifiers in the not too distant future by a simple
- The AuthorityId string "ADS" has been specified. This simply
recognizes the current role of ADS in managing the namespace used for
these identifiers, in the
absence of a community-wide namespace granting authority. It does not
suggest nor imply that ADS controls or manages the dataset itself.
- The ResourceId token will be interpreted as a Facility. An
ever-growing list of facilities is
maintained by ADS. Data centers should contact ADS should they need
to register new entries.
- The PrivateId string can be anything that the data center
desires, with the provision that the identifiers string as a whole
should abide by
the general syntax of a URI,
as required by the IVOA identifiers specification.
Generating Dataset Identifiers
All Data Centers and Archives which provide public access to their
data should structure their databases and interfaces so that when a
particular dataset is released to the public, it is uniquely tagged by
an identifier ID created as discussed above. Users who download one
of such datasets should be made aware of the identifiers associated
with it and how it should be referenced in the published literature.
In order for a datacenter to ensure that the identifiers it is
generating comply with the syntax endorsed by the ADEC, the following
- The identifier is in the form
FacilityId has been registered with
ADS and is listed in the
table of known facilities
PrivateId is a unique identifier within the
FacilityId, and its association with the dataset will not change.
- A profile for the datacenter has been registered with the
ADS, and in it
FacilityId has been listed as
one of the resources that the center has data for.
- The datacenter provides a dataset verification
service which will be used to verify the validity and location
of identifiers published in the literature.
Once a datacenter has published a dataset ID, it should provide access
to it. Ideally this will be a human-readable page on its web server
displaying the dataset's relevant metadata and offers the user the
option to download the dataset itself in some form or fashion. It is
left up to the datacenter to decide what to do if and when a revised
version of a particular dataset is published. In general, however,
it is understood that access to the latest revision of a dataset should
be an option if not the default.
Registering a Datacenter Profile
In order for ADS to coordinate the verification and linking of dataset
identifiers to the appropriate datacenters, it is necessary for the
datacenters to provide some basic metadata about its data holdings
and services. While it is expected that the appropriate metadata will
one day be made available by a public VO registry, its format and
access methods are at this time not available. As an intermediate
solution to the problem, we require that the data centers maintain
a simple profile which will provide ADS with the necessary metadata.
The data center profile is simple XML document that lists the
data center name and description, the name and email address of
the person responsible for the maintenance of the profile,
the URL of the web service to be used for dataset verification,
and the list of facilities that the datacenter has data for.
For more information, please see
this simple example.
Two options are available for creating and maintaining such a profile
Once the datacenter profile document has been created, the person responsible
for it should let ADS know
of the profile's location by submitting its URL to the
ADS will review the profile and merge it into its list of datacenters
to be used for the verification of dataset identifiers. Also, ADS will
periodically harvest the URL corresponding to the datacenter's
profile and will update its list of datacenters and supported facilities
accordingly. Once registered, a datacenter can update its profile
without further intervention from the ADS.
- Create the appropriate XML document and make it available
at a stable URL on your web site.
- Install the ITWG SOAP data verification toolkit that has a built-in
option to generate such a profile when invoked with the proper syntax
(please see below for more details).
Providing Data Verification Capabilities
In order to promote an open framework that can be used for the
distributed verification of dataset identifiers across data centers,
the ADEC ITWG (Interoperability Technical Working Group) has created the
specification for a SOAP-based web service. The corresponding
WSDL file can be used to
generate client and server interfaces to the service.
providing data verification services should provide and maintain a
service that abides by this specification.
ADS provides a central verification service that fans out queries
to the appropriate datacenters. A diagram showing the architecture
of the system and the datacenters currently providing the verification
services (as of mid-2003) is
To facilitate the deployment of verification services, the
ADS also developed a
that greatly simplifies the
creation of a compliant web service.
Among other things, by defining a few variables and installing a simple CGI
script based on this toolkit you will be able to automatically define your
site's profile described above.
For more information, please see the
What follows is a list of additional resources and information
regarding this effort.
Please bear in mind that this is work in progress, so some of the
things listed below may not be (yet) working the way they are intended
The following resources are mostly of interest to the Data Centers
- A PERL SOAP Toolkit
that can be used by Data Centers to set up a dataset verification
service. The README file
contains some information about installation and deployment
- The WSDL file for the ADS test
SOAP service that performs dataset verification. Each datacenter
providing data verification services should have a similar profile
(except of course for the location of the service itself).
A number of things still remain to be done:
- Bring on-line verifiers for all NASA centers.
- Enhance the central data verifier by threading the fan-out queries.
- Have the ADS central data verification script cache the
verification results so that we don't need to constantly query all
datacenters for every verification task.
- Create a simple data verification resolver that can act
as a central proxy to all the instances of a verified dataset, as
described above. This is a trivial task once the caching of results
- Update ADS's central dataset verification service so that we can use a
standard API for it (I'm thinking of using the same SOAP service that is
used to verify identifiers with the individual data centers).
Obviously, this needs to be worked out between ADS, UCP and other
publishers and can become a simple system upgrade once all the other
pieces are running.
If you have comments or questions about the contents of this document,
please send me an email.
Alberto Accomazzi -- Last modified: 14 Nov. 2004