In 2005, CRL was awarded funding by the Andrew W. Mellon Foundation [1] to develop the procedures and activities required to audit and certify digital archives. The project began in May 2005 and continued through January 2007.
Rigorous assessment is necessary to determine the level of assurance a particular archiving service provides to publishers and users, and to ensure that the valuable digital resources archived, such as electronic journals, news, and other scholarly content, will continue to be available and functional. Certification will enable universities, libraries, publishers, consortia, and others to protect their investments in developing and securing access to knowledge and evidence in electronic form, and thereby foster confidence in digital archiving.
The project built upon the work of the [2]RLG-NARA Digital Repository Certification Task Force [3]to develop digital archives certification processes and metrics. By testing the RLG-NARA metrics through actual audits of three digital archives and one archiving system, project staff developed a detailed set of methodologies for auditing and certification, and identified the corresponding costs.
The project also laid the foundation for CRL’s subsequent case studies of Long-Lived Digital Collections, a project supported by the National Science Foundation.
Millions of dollars are invested annually by universities and research organizations, public and private, in subscriptions to electronic journals and databases. Unlike investments in traditional library collections, electronic subscriptions secure access to resources that are “hosted” elsewhere, often by third parties, outside the control of the subscribing institution.
Many publishers and other producers of content also entrust the management of their databases, online information, digital collections, and other digital assets to independent, third-party organizations. Even heritage materials in electronic form, such as news broadcasts and legal documents, are now managed and archived in ways that challenge conventional ways of ensuring that library collections will be available to future scholars. In addition, many countries have legal requirements for the deposit of digital materials that must be addressed by both libraries and publishers (see www.nla.gov.au/padi/topics/67.html [12] for more information on legal requirements for the deposit of digital materials.)
Given the scale of such investments, the reliability of archiving institutions and their systems and processes is critical. It is important to be able to determine that an issue of an electronic journal or newspaper will be maintained by the archiving entity and will be able to be presented, with all important characteristics intact, at a future date.
Although accepted practices exist for the valuation and auditing of other kinds of assets, few means enable libraries and publishers to assess and manage the risks they incur in investing in access to electronic knowledge resources. Such investments cannot be subjected to the same kind of auditing and validation as other financial investments of a comparable scale. In the absence of reliable certification, moreover, libraries and publishers now subsidize costly fail-safe strategies that involve the warehousing of voluminous and redundant print holdings.
CRL's Certification of Digital Archives project drew upon the findings of the RLG/NARA Task Force to specify in detail the criteria to be applied in the test audits.
Project advisors reviewed and critiqued the proposed auditing methodology and instrument. The advisors then determined that the process outlined provided data useful to decision-making about the trustworthiness of particular archiving arrangements, the disposition of digital scholarly and heritage content, and investment in access to such content archived in particular repositories.
To test the metrics and certification process, project staff conducted test audits of three repositories of digital content maintained by third-party organizations:
This undertaking drew from knowledge and best practices in the fields of digital preservation, library and museum management, database indemnification, disaster preparedness, and university and cultural heritage institution financial management.
Bernard F. Reilly [17], President of the Center for Research Libraries
Reilly has been an active leader in CRL preservation initiatives and has managed digital library projects and digital resource licensing and contract arrangements in the for-profit and non-profit sector for more than ten years. As principal investigator, he oversaw the work of the project director, and led the deliberations of the panel of project advisors.
Robin Dale, Program Officer, Research Libraries Group
Dale was the cochair and principal investigator of the RLG-NARA Task Force on Digital Repository Certification, which finalized its draft set of certification standards and criteria in 2006. The Task Force provided vetting and advice to the project, supporting and augmenting the work of the project advisors.
The project advisors brought to the effort knowledge and insights from the fields of scholarly publishing, digital preservation, e-journals licensing, fiscal management and oversight, and information technology administration in higher education.
A systems analyst, project technician, and technical consultants also supported the effort.
The project will leverage the work of the RLG-NARA Task Force on Digital Repository Certification. The group identified certifiable metrics related to attributes, processes, functions, and activities of a digital repository or types of repositories and also designed a standard certification process framework that can be implemented across domains or types of digital repositories. This framework, together with pertinent metrics developed by other communities, will be the basis for the metrics used in the CRL test audits.
Project staff will develop reporting terminology for codifying and expressing the results of the audits and certifying the archives. The reporting terms will cover the full range of aspects of the archives audited.
The terms will then be indexed to a rating or certification scheme devised for digital archives. The scheme will reflect the full range of criteria applied in the auditing process. Because not all electronic resources are equally valuable or costly, and because different types of content require different types and levels of archiving solutions, all archives will probably not be certifiable to a single standard. A range or scale of possible standards, each with a corresponding specified level of assurance and cost, will be devised.
Project staff will also determine the frequency at which certification of a single archive should be undertaken or renewed. Archives that hold dynamic or rapidly growing content will have to be audited more frequently than archives of relatively static content. Again, the audit cycle will vary according to the value of the content, cost of the resource, and needs of its users.
The purpose of auditing a digital archive is to determine the degree of assurance the archive provides for the long-term availability and functionality of the digital resources maintained therein. The audit must establish with a high degree of certainty that an archive can, at a given point in time, present content that is deposited in or ingested by the archive, and that said content will retain its critical characteristics and functions.
Project staff will conduct audits, analyzing and evaluating the various activities and processes of each of the subject archives through on-site visits to the archives, examination of pertinent specifications and documents, interviews, and periodic queries. The audit may include analysis of audited financial statements of the publishers and archiving organizations involved.
The project advisors will then analyze the results of the audit, including test materials and test outputs, and determine the usefulness and completeness of the audit and resulting data. The auditing methodology will then be revised as appropriate.
A certifying organization or agency, capable of sustaining itself, will be needed to provide certification services to the field on an ongoing basis. At minimum, such an agency will have to ensure accountability to the research, publishing, and cultural heritage communities. Project staff and the principal investigator will identify the essential functions of such an agency and the major features of an appropriate business model to support it. At minimum, such a model must possess an organizational structure, governance, and funding system that ensure accountability to the higher education and scholarly communities. Ideally, the optimal certifying agency will have a large and diversified client base, to ensure that no single client or stakeholder can exert undue influence on its policies and activities or compromise its accountability to the research community.
Project staff will present and analyze costs of the auditing and certification activities on the basis of the subject archive audits, and identify benefits and assets accrued. They will then identify the range of potential stakeholders and “markets” for certification services and products, and quantify the prospective value of those products and services to each market.
The project director, principal investigator, and advisors will examine a number of existing models for certifying and rating organizations in the for-profit and nonprofit fields, analyzing their respective funding systems or payment models, organizational structures, and forms of governance.
This is the working schema that CRL applies in assigning summary ratings of a repository’s compliance with the TRAC criteria. The numeric rating is based on a scale of 1 through 5, with 5 being the highest level, and 1 being the minimum certifiable level.
5. Compliant with all metrics fully and consistently, and able to provide complete, up-to-date documentation of all systems and procedures and certifications of system security.
4. Compliant with all metrics, and able to provide complete, up-to-date documentation, but with minor inconsistencies in areas that are not likely to lead to systemic or pervasive defects.
3. Compliant with all critical metrics, and able to provide complete, up-to-date documentation of major systems and procedures, but with minor inconsistencies in areas that are not likely to lead to systemic or pervasive defects.
2. Compliant with all critical metrics, with a minimum of inconsistencies in areas that might lead to minor defects of a systemic or pervasive nature; documentation is complete and updated on a periodic basis.
1. Compliant with all critical metrics, with a minimum of inconsistencies or deficiencies in areas that might lead to minor defects of a systemic or pervasive nature.
Links
[1] http://www.mellon.org/
[2] http://www.oclc.org/programs/ourwork/past/repositorycert.htm
[3] http://www.oclc.org/research/activities/repositorycert.html
[4] https://www.crl.edu/archiving-preservation/digital-archives/past-projects/cda/background
[5] https://www.crl.edu/archiving-preservation/digital-archives/past-projects/cda/personnel
[6] https://www.crl.edu/archiving-preservation/digital-archives/past-projects/cda/timetable
[7] https://www.crl.edu/archiving-preservation/digital-archives/past-projects/cda/metrics
[8] https://www.crl.edu/archiving-preservation/digital-archives/past-projects/cda/participants
[9] https://www.crl.edu/archiving-preservation/digital-archives/past-projects/cda/outputs
[10] https://www.crl.edu/facets/archiving-and-preservation
[11] https://www.crl.edu/reports
[12] http://www.nla.gov.au/padi/topics/67.html
[13] https://www.crl.edu/sites/default/files/d6/attachments/pages/KBprofile_0.pdf
[14] https://www.crl.edu/sites/default/files/d6/attachments/pages/portico_0.pdf
[15] https://www.crl.edu/sites/default/files/d6/attachments/pages/ICPSR_0.pdf
[16] http://lockss.stanford.edu/
[17] mailto:breilly@crl.edu