Designing for Automated Digital Preservation : Model, Pre-Ingest, and Error Handling

Abstract: With the rapid increase in the amount and complexity of data that is needed to be preserved, manual preservation activities produce complex, lengthy, and costly processes. Therefore, automation of preservation processes, together with modeling of workflows and streamlining, can help reduce costs and enhance the focus on preservation processes.  Accordingly, the research question is defined as: “How to establish an automated many-to-many interaction between Information Systems and digital preservation systems?”This research proposes a model and instantiation of middleware as a standalone system, which could be hosted in the cloud, for bridging between ISs and DPSs including three sub-parts making both many-to-many capacity and automation of interactions possible: pre-ingest workflow, Context-aware Preservation Manager (CaPM), and error-handling workflow. A Design Science Research (DSR) approach was taken to conduct this research consisting of three design cycles to design and develop each of the three sub-parts of the solution artifact, i.e. the middleware.  The middleware consists of several action-based components and an administrative component (CaPM) which carries out the automation of the tasks in the middleware. The action-based components are designed to complete a pre-ingest workflow to prepare digital content sent from an information system to be transferred into a digital preservation system. The path for the pre-ingest workflow, i.e. which components are going to process the digital content and in what order, is automatically defined by CaPM according to the information system’s preservation policies. Standard interfaces are used for middleware’s internal or external communications to promote its scalability in the long run as well as its capability of embedding additional workflows or processes developed in the future, e.g. post-access workflow.An additional outcome of this research is proposing five design principles aiming to contribute to the knowledge for future design practices: DP1. Provide rule-based definition of workflow execution path so that the middleware affords IS to implement their preservation policy and metadata extraction requirements. DP2. Provide capability of executing alternative workflow routes so that the middleware affords IS to ensure a successful encapsulation and submission of SIP. DP3. Provide features for gathering preservation data in the middleware so that the middleware affords preservation planning support. DP4. Provide an automated error-handling workflow with compensating action so that the middleware affords to minimize manual intervention in case of errors in a workflow. DP5. Provide capability of executing concurrent workflows so that the middleware affords IS and DPS many-to-many interactions via the middleware.The results of this thesis contribute to the state-of-the-art in a few aspects:Compared to existing solutions, such as pre-ingest tool developed for Finnish National Archives and UAM for Estonia, that need to be installed on a user’s system, integration with the middleware is carried out with less complexity. This is achieved by designing the middleware as a standalone system that could be hosted in the cloud along with using standard communication interfaces, which further make the middleware adaptable to changes or upgrades in the environment it operates in. Such capability of the middleware in handling many-to-many interactions goes beyond what was introduced in previous middleware architectures for Digital Preservation System’s integration with Information Systems.The middleware solution for pre-ingest in this thesis, in comparison with the similar recent solutions, promotes automation capabilities especially for preserving complex digital content (e.g. databases, workflows), automatic execution of the pre-ingest workflow, or in case of a need for using multiple external digital preservation solutions or services.CaPM monitors the execution of workflows and can update or abort a workflow path if needed. An aborted workflow caused by an error/failure will automatically be replaced by an error-handling workflow with compensation action, hence increasing the level of automation. Automation of such functionalities, as well as the approach for handling errors, has not been applied in previous tools.CaPM can also contribute to the current stream of research on decisions making regarding preservation planning and strategies by providing logged data about the digital objects passing through the middleware.While the solution artifact of this research provides middleware to perform as a bridge for automated many-to-many interactions between information systems and digital preservation systems, the resulted design and implementation of the middleware components cover only one direction of such interaction, from information system to digital preservation system (pre-ingest).