A Study on Architecture of Massive Information Management for Digital Library

(Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China) 

XING Chun-Xiao+, ZENG Chun, LI Chao, ZHOU Li-Zhu 

Abstract

This paper investigates the challenging issues and technologies in managing very large digital contents and collections, and gives an overview of the works and enabling technologies in the related areas. Based on the analysis and comparison of the related work, a novel architecture of massive information management for digital library is designed. The key components and core services are described in detail. Finally, a case study THADL (Tsinghua University architecture digital library) that complies with the architectural framework is presented. 

Key words: digital library; architecture; massive information management; interoperability; metadata

 

1 Introducn 

In the recorded hitiostory of human being, the printed materials used to play a dominant role in the preservation and pervasion of human information and knowledge. However, with the rapid development of technologies in computer, communication, multimedia and storage, this role is giving away to the digital resources in the new era. The explosive growth of information in digital forms has posed challenges not only to traditional archives and their information providers, but also to organizations in the government, commercial and non-profit sectors. According to the latest report by Lyman and Varian, the world’s total yearly production of print, film, optical, and magnetic content would require roughly 1.5 billion gigabytes of storage which is roughly 250 megabytes for every person on the earth. Printed documents of all kinds comprise only 0.003% of the total. Magnetic storage is by far the largest medium for storing information and is the most rapidly growing section, with a shipped hard drive capacity doubling every year. The types of digital resources are diverse. They include digital texts, documents, scientific data, images, animation, video, audio etc. The applications of the digital resources are quite broad, including DL (digital library), movie/video center, other public media (television, broadcast, newspaper, etc.), museum, and national or cooperative information center. At the same time the information highway, which is represented by Internet, has been an important tool of the pervasion of digital resources. The governments, companies, groups, research institutes, non-government organizations, education institutes all over the world put massive information on the Web.

 

Technology challenges and key issues 

These massive digital resources present many challenging issues in data management technology area. The following are some examples. 

Data model. 

Traditional data model theories are only applicable to structured data, but not for the massive digital resources of various types and they are mostly semi-structured or unstructured. Thus, new data models are demanded.

System architecture. 

Traditional database management systems are designed for business data processing featured by concurrent, short, and update transactions. Therefore transaction management and concurrent control remains as the center of system architecture. The architecture is not suitable for the management of digital resources as classical transaction concept is becoming less important in these resources. We need to pursue novel and universal frameworks for massive digital resources management.

Massive information storage. 

The volume of digital data resources is counted by terabytes or petabytes. Traditional storage devices using SCSI cannot work for efficient storage, online migration and persistent archive of such massive digital resources. So the research of multi-level storage systems, SAN (Storage Area Networks) and other technology are inevitable.

 

Query processing. 

In traditional database systems, queries are expressed in query language such as SQL, but in the query and search of massive digital resources, many new mechanisms should be used, such as keyword search, full-text search, similarity query, and content-based multimedia retrieva l. How to integrate the query methods (including SQL, OQL, and different XML query languages, e.g., XQL, XML-QL, XML-GL) efficiently to build an efficient and flexible query processing method has not been satisfactorily solved yet. 

To solve the problems mentioned above will remain as a major goal to researchers in the next few years. To fulfill this end, we present a novel architecture for massive information management of digital resources in this paper. This architecture is intended to meet the requirements of managing digital resources characterized by distributed, dynamic, massive and heterogeneous properties.

 

2 Overview of the Related Work 

The IEEE STD 610.12[2] defines architecture as the structure of components, their relationships, and the principles and guidelines governing their design and evolution over time. A wealth of previous work has addressed the research and development of architecture for digital library. Now we will give an overview of the related standards and enabling technologies as follows.

 

Digital library formal model..5S model 

Digital libraries are complex information systems and therefore demand formal foundations lest development efforts diverge and interoperability suffers. Reference [12] proposed the fundamental abstractions of 5S (streams,structures, spaces, scenarios, and societies), which contribute to define digital libraries rigorously and usefully. Streams are sequences of abstract items used to describe static and dynamic content. Structures can be defined as labeled directed graphs, which impose organization. Spaces are sets of abstract items and operations on those sets that obey certain rules. Scenarios consist of sequences of events or actions that modify states of a computation in order to accomplish a functional requirement. Societies comprehend entities and the relationships between and among them. Together these abstractions relate and unify concepts, among others, of digital objects, metadata, collections, and services required to formalize and elucidate digital libraries. The formal model shows it can clearly and formally define a minimal digital library. But the formal model still needs to be improved and revised in digital library development.

 

3 Architecture of Massive Information Management for Digital Library 

Until recently, there has been no common approach for architecture development and use in large-scale digital libraries construction. All kinds of libraries, research institutions and universities traditionally developed their DL architectures using techniques, vocabularies, and presentation schemes that suit their unique needs and purposes. In this section, we propose a new architecture of massive information management for digital library based on analyzing and researching the related works. We design the architecture by complying with related standards and making use of enabling technologies. The motivation is providing a common framework and software platform for constructing the large-scale digital library.

 

3.1 Design principles

The following set of principles for building architectures are critical to the objectives of the guidance. (1) Standardization. We will comply with related international and national standards in all layers of the architecture. (2) Componentization. Because it is widely accepted as a good software engineering practice, most modern programming environments adopt some form of component models. (3) Reusability. The technical infrastructure that provides functional specifications, paradigms of object-oriented programming, and component model has been particularly effective strategies for producing reusable and powerful software. (4) Scalability. The scalability of the proposed architecture is achieved by adopting a progressive and multi-layer framework, and providing the mechanisms for scaling services appropriately based on the service-demand and resource-availability. (5) Interoperability. Interoperability among heterogeneous systems and collections is a central theme. The different systems and collections have a wide variety of data types, metadata standards, protocols, authentication schemes,and business models. The goal of interoperability is to build coherent services for users by integrating components that are technically different and managed by different organizations. This requires agreements to cooperate at three levels: technical, content and organizational levels. (6) Architectures should be relatable, comparable, and scalable across DLs. This principle requires the use of common terms and definitions, such as metadata, digital object, repository, collection, and so on. This principle also requires that a common set of architectural building blocks is used as the basis for architecture descriptions. (7) Architectures should be adaptable, reliable, maintainable and testable. The ultimate achievement of this principle will reduce overall software costs, improve product and service quality, help organizations to thrive, or even survive in our current and future turbulent times.

 

3.2 Architecture design

We design a multi-layer architecture to support the service and management in DL (digital library) based on complying with a set of design principles above. The architecture is a more OSI-like reference model, but it focus on the massive information service, management, and archival.

 

3.3 Key functional components

Data ingest.

This component provides the services and functions to accept resource digitalizing, marking, indexing, and cataloging from producers, and prepares the contents for data management and knowledge management. Data ingest function also performs quality assurance on metadata and digital objects, which complies with the related data formatting and documentation standards. Meanwhile Data Ingest also gathers and catalogs new metadata and digital objects from remote DLs and WWW by using interoperability protocols.

 

Data management. 

The component provides the services and functions for storing, maintaining, and accessing both metadata repositories and collections from data ingest. Data Management functions include administering the metadata database, multimedia databases and file systems. It will maintain schema, view definitions and referential integrity, and perform database updates (loading new descriptive information or administrative data). The central storage of metadata is provided by data ingest and metadata is gathered from other DLs by interoperability protocols. By using output interfaces between layers, it will provide data services to service layer, such as search and browse services. Meanwhile, object server (such as video server) that is managed by data management provides all kinds of multimedia services such as VOD. Data is typically stored as digital objects in file systems or as blobs in object-relational or object-oriented databases. Descriptive metadata is typically stored as attributes in metadata databases that complies with metadata standards, such as Dublin Core, USMARC, CNMARC. The metadata attributes use the XML (eXtensible markup language) to label the information content and a DTD (document type definition) to build a semi-structured representation of the information.

Knowledge management.

Knowledge is represented as sets of relationships of domain concepts that are expressed as rules used within inference engines. This component provides mechanisms for managing all of these types of relationships. Every discipline deals with multiple concept spaces that describe relationships between physical variables, data sets, collections, applications, and domain knowledge. These concept spaces provide a hierarchy of levels of the implied knowledge based on ontology. Ontology[15] is the key technology used to describe the semantics of information exchange, which is defined as specifications of a shared conceptualization of a particular domain. They provide a shared and common understanding of the domain that can be communicated across people and application systems, and thus facilitate knowledge sharing and reuse. The concepts and their relationships as knowledge rules are stored in knowledge repository. The components will provide services for the manipulation of specific applications.

 

Data archival. 

This component provides the services and functions for the storage, maintenance and retrieva l of metadata, digital objects, and knowledge rule for the long-term preservation. The functions include receiving the data need to be archived from data ingest and data management components, and adding them to the permanent storage, managing the storage hierarchy, refreshing the media on which archive holdings are stored, performing the routine and special error checking, and providing the data migration, replication, backup, and disaster recovery capabilities. The system will adopt different networked storage methods (such as NAS, FC-SAN and IP-SAN) according to different system scale and application requirements. The data archival standard OAIS reference model will be used in the architecture for the understanding and increased awareness of archival concepts needed for the long term digital information preservation and access. Meanwhile, we extend the model and make it suitable for archival requirements of digital library.

3.4 Core services components

(1) Discovery and search: The components are to provide fundamental capabilities for locating and finding resources and collections among the distributed digital libraries. The challenge is to encourage end-user resource discovery and information use in a variety of formats, from a number of local and remote sources, and in a seamlessly integrated way. The architecture uses metadata for resource discovery, and uses a keyword indexing search-engine for resource discovery as a complementary method. The standards and specifications for resource discovery include Z39.50, OAI, and SDARTS. The key techniques include multi-agent and middleware technologies that are based on metadata including USMARC, CNMARC, and Dublin Core Element Set 1.0. 

(2) Content-Based retrieva l: The components are to provide fundamental capabilities for query multimedia resources and collections, including text, image, video and music. The content-based image retrieva l allows for image queries based on image examples, feature specifications, and primitive text-based search. Content-based video analysis, automatic video index, summarization, and relevant feedback are used for the video retrieva l. 

(3) Personalized service and notification: To quickly and easily gather useful information and knowledge to alleviate information overload problem, it has therefore become necessary to provide users with active and adaptive service mechanisms that automatically extract only relevant incoming documents. The component is able to provide the users with a personalized filtering and notification service based on user modeling and profile learning. 

(4) Right management and payment: The first-generation of DRM (digital rights management) focuses on security and encryption as a means of solving the issue of unauthorized copying. That is, lock the content and limit its distribution to only those who pay. The second-generation of DRM covers the description, identification, trading, protection, monitoring and tracking of all forms of right usages over both tangible and intangible assets, including management of right holder relationships. 

4 Case Study: THADL

Tsinghua University Architecture Digital Library (THADL) is developed as a prototype system, which started formally since March 2000. THADL maintains a balance between technology-focused research and content-based research. The project team brings together three research groups from different disciplines including computer science, architecture science, and library information management, and represents a substantial cooperation among computer researchers, librarians, and subject specialists. The large amount of multimedia materials in THADL repositories include papers, journals, photographs, manuscripts, drawings/blueprints, animation, video, and audio on Chinese ancient architecture. We have finished following the goals of THADL. By designing and developing THADL prototype, we analyze and summarize the previous architecture. Meanwhile the architecture also guides the improvement and extension of system functions and services. The detail of THADL has already been described. The main research contents of THADL are as follows: 

(1) Exploring the efficient methods and technologies by analyzing, designing, and eva luating the THADL prototype system to pave the way for constructing a future large-scale digital library;

(2) Building a Chinese architecture digital library that provides an intelligent, interactive, and collaborative learning environment on the Internet rather than the static digital resource repositories; 

(3) Presenting an efficient method to digitalize, index, and preserve most kinds of materials for Chinese architecture study; 

(4) Establishing metadata specifications and standards and their related issues for Chinese architecture science; 

(5) Supporting friendly, active, and personalized services for different users including students, scholars, librarians, and ordinary users on the Internet. However, THADL is a medium but comprehensive prototype system, and we have a long way to go for researching, testing, and analyzing whether our proposed architecture is suitable for the large scale digital library application such as the China Digital Library Project.

 

5 Conclusion and Future Work 

In this paper we discuss the challenging issues and technologies in managing very large digital contents and collections, and give an overview of the related works and enabling technologies for supporting high performance data-intensive applications. We design a novel architecture of massive information management for digital library, and describe the key components and core services. Finally, the case study..THADL (Tsinghua University Architecture Digital Library) is given according to the architectural framework. 

In the future work, we will study and develop software middleware for massive storage management, XML based search engine, and multilingual full-text search. We will design and implement a lightweight interoperable protocol based on the tailed Z39.50 protocol for supporting the large-scale distributed heterogeneous digital resources integration. 

References: 

[1] Baldonado M, Chang CK, Gravano L, Paepcke A. The Stanford digital library metadata architecture. International Journal on Digital Libraries, 1997,1(2):108~121.

[2] Lagoze C, Hoehn W, Millman D. Core services in the architecture of the national digital library for science education (NSDL). In: Proc. of the 2nd ACM/IEEE-CS Joint Conf. on Digital Library. Portland: ACM Press, 2002. 58~65.

[3] Wactlar H. Multi-Document summarization and visualization in the informedia digital video library. In: Proc. of the 12th New Information Technology Conf. Beijing: Tsinghua University Press, 2001. 323~332.

[4] Gon?alves MA, Fox EA, Watson LT, Kipp NA. Streams, structures, spaces, scenarios, societies (5S): A formal model for digital libraries. Technical Report, TR-01-12, Virginia Tech, 2001.

[5] Thornburgh RH, Schoenborn BJ. Storage Area Networks: Designing and Implementing a Mass Storage System. Prentice Hall, 2000.

[6] Clark T. IP SANS: An Introduction to iSCSI, iFCP, and FCIP Protocols for Storage Area Networks. Addison-Wesley, 2001.

[7] Staab S, Studer R, Schnurr HP, Sure Y. Knowledge processes and ontologies. IEEE Intelligent Systems, 2001,16(1):26~34.

[8] Xing CX, Wu KH, Luo DY, Zhou LZ, Liu GL, Qin YG. THADL: A digital library for Chinese ancient architecture study. In: Proc. of the 12th Int'l. Conf. on New Information Technology. Beijing: Tsinghua University Press, 2001. 373~382.