Metadata Harvesting
nzresearch.org.nz is a database containing copies of the document metadata from Institutional research Repositories around New Zealand. This database is built up and maintained through a process called metadata harvesting.
The metadata database is updated every night by a fresh harvest. We harvest the metadata using the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH), which lets us request only those records that have changed since the last time we asked, so that we do not have to make a complete copy of the database every night. Most nights only a handful of records are updated. We do a full harvest of the repositories once every month or so.
The Metadata Guidelines
In order to provide features like Browse by Author, Browse by Subject, and filtering by document type in search for all the research documents in the database, we first have to make sure that the metadata in all the different repositories is in a similar, known format.
To encourage this consistency, and to encourage institutions to provide the best possible metadata, the project collaborators have agreed to follow a set of metadata guidelines as closely as possible. These metadata guidelines are available from the National Library of New Zealand document repository
Metadata Verification and Transformation
We currently harvest metadata in Dublin Core format [link] (specifically, we use the oai_dc encoding for OAI-PMH). However, simple Dublin Core metadata is not always the best format for use in the Research new Zealand web interface. We therefore use metadata verification and transformation to make the metadata more useful.
Metadata verification is the process of checking the harvested metadata to make sure it is complying with the metadata guidelines. We do this by applying an XSL transformation to each metadata record as it is read, which is used to generate a set of metadata errors and warnings. These errors and warnings are stored as NZIR Administration (nzir_admin) metadata and attached to the records in nzresearch.org.nz.
Note that most of the records that are harvested have no errors or warnings, which means they don't have any NZIR Administration metadata.
Once the metadata records have been verified, we use another XSL transformation to convert them into a consistent format that we can use to implement advanced search and browse features. We call this NZIR Internal (nzir_internal) metadata, and almost all records have it (you can access it from the metadata record screen).
These metadata fields are generated for each record:
- Title: a single clean title used in Browse By Title [link].
- Author: a list of Authors in .Last-name, First-names. format used in Browse by Author [link].
- Year: the Year of publication used when citing this material.
- Link: the URL of the document.
- Marsden Code: a list of subjects (topics) that this record is relevant to, encoded using Marsden Fields of Research codes. This metadata is generated from harvested subject metadata in many formats.
- Type: a single type in Eprints Type Vocabulary format. This metadata is generated from harvested type metadata in many formats.
Metadata quality reports
Every month, nzresearch.org.nz produces metadata quality reports to assess the state of the metadata in New Zealand Research Repositories.
These reports show the number of metadata records that have been harvested, and the number of errors and warnings that were reported during the metadata verification process. They are also broken down by institution.
You can find access the metadata quality reports from the Reports pages.
