[SJ Logo]Annual Report to eLib

Home | Search | Demo | News | Feedback | Members Only

SuperJournal Project, December 1995-July 1996

1. Introduction

SuperJournal is a project in the Electronic Journal division of the eLib Programme. Its goal is to learn the factors that will make multimedia electronic journals successful: what features and functionality of electronic journals have real value to readers and authors. This involves the creation of knowledge and the sharing of knowledge in the stakeholder community so that it enables change.

The SuperJournal project began work in December 1995 and has therefore been operational for 8 months. This Annual Report describes the objectives we have set for year 1, the activities we have performed to achieve them, and what we feel our successes have been. In reviewing this report, we ask that readers keep in mind that a project to create knowledge that will enable change is fundamentally different from a project to build a system and develop it into a self-sustaining service. The evaluation research is different and the project self-evaluation is different. In this respect SuperJournal is different from most of the eLib projects.

In a system-creation project, the aim of evaluation is to assess the system so it can be improved, and the best system possible can be delivered to users. In a knowledge-creation project, building a system is an instrument to create knowledge about what should be delivered, ie it is a means to an end. The system will not have all that is required and may not achieve great utility or use. However, it provides a forum within which readers and authors can make informed judgements about electronic journals and the features which have real value. The evaluation assesses what use was made of the system offered and why, and how it could be changed to give better knowledge.

2. Objectives and Targets

The following project milestones have been agreed with JISC (the wording has been updated to reflect current terminology within the project):

Timing Date Milestone
5 months May 1996 Project Plan
8 months August 1996 Prototype application, one journal cluster
12 months December 1996 Working application, one journal cluster, one user site online

To achieve these milestones, the following objectives for 1996 were set:

The next section outlines the activities we have undertaken to achieve these milestones and objectives. We feel all objectives have been met, or are on schedule to be met as the year progresses. Only the first milestone falls within the timeframe of the Annual Report: the Project Plan. This milestone has now been achieved, but was not achieved on the agreed date.

3. Major Activities

3.1 Project Planning

The first task was to establish a project office and to hire the necessary staff. A Project Manager was appointed on 4 December 1995, the formal start date of the project. Manchester Computing hired a Technical Project Manager, a Data Handler, and a Developer, and HUSAT allocated research staff. By 1 February 1996 the project was staffed and the project office set up.

The next task was to develop a project structure:

The Planning Committee's first task was to agree project milestones, deliverables, project tasks, and a schedule for year one. In April a Project Plan was circulated to the Planning Committee, to provide a framework for the project work, outline the roles and responsibilities of the large number of institutions and individuals involved, and explore the issues that will be important for the project's success.

3.2 Publicity

In January and February a number of activities were undertaken to publicise the project:

In January a brochure describing the project was produced and circulated widely, both within the project and outside. It has proved a useful communication tool to gain support for the project within the institutions involved, eg for publishers to describe the project to their journal editors and authors, or libraries to describe the project to their faculties. Associated with the brochure was the creation of a project identity and logo.

In February Manchester Computing developed a Web site for SuperJournal at http://www.superjournal.ac.uk/sj/. This includes “public” information about the project, and will include results as they are achieved. The Web site also provides access to “private” Web pages for project participants for work-in-progress. In June the project also established an email discussion list for project participants.

Several presentations were given at various meetings, and articles were published, in traditional and electronic journals. Manchester Computing featured SuperJournal at their Open Days on 22-23 March 1996.

3.3 Kickoff Meeting

A Kickoff meeting was held on 26 March to formally launch the project, and all project participants were invited. The purpose was firstly to make sure everyone involved understood the project, its objectives, timescale, and the roles of participants. Secondly, it enabled everyone to meet each other, make personal contact, and explore the issues which would make the project a success from their individual perspectives.

It was considered a great success by all, in particular a session where the publishers and libraries outlined key issues for the electronic future as they see it, the issues they feel are important, why they are participating in SuperJournal, and what they hope to learn from the project.

3.4 Desk Research on Electronic Journals

To inform us in developing the Evaluation Plan, HUSAT did two “state of the art” literature reviews. The first is called “Factors Determining Usage of Electronic Journals” and reviews the content and results of previous studies on electronic journals. It focuses on what has been learned to date from the reader's perspective. The second is called “Evaluation Methods for Electronic Journal Research” and reviews the relevant human factors methods, highlighting strengths and weaknesses, to assess suitability for use in the project. Both reports were completed in May.

3.5 Project Research and Evaluation

The purpose of the project is to identify the features and functionality of electronic journals that have value to readers and authors, ie what are the unique selling points of a multimedia electronic journal. Unique selling point (USP) is a term used in marketing, but we have found it useful shorthand to refer to those critical value-added features that will make electronic journals sufficiently compelling that readers and authors will want to use them, and even prefer electronic to paper. In the end, the project should be able to say what these features should be, and the priority that can be attached to individual features in building an electronic journal of value to the research and teaching community. To this end, the purpose of the research is to identify what is useful, rather than what is usable.

The first task was to develop an evaluation strategy. In April, following guidelines developed by Tavistock, HUSAT developed an evaluation strategy which is:

A number of activities were involved in developing the evaluation strategy into an evaluation plan. Firstly, we needed to agree on the specific questions the research should address. In April a long list of research questions was drafted, with questions organised by stakeholder. This was circulated widely in the project for comment, discussed, refined, and finally reduced to a manageable number of important questions the research will address.

In June a paper was drafted summarising the objectives of the project research, along with a list of features that are likely to form the USPs of multimedia electronic journals. This paper, the list of features, and the research questions allowed us to focus on the relevant research activities and data to be collected. Importantly, we decided that the research should address author as well as reader needs. The features that are USPs from the reader's point of view have implications for authors, the content they include in their papers, and the editorial process. We need to consider the tools and standards that will facilitate this, and the USPs from the author's point of view that will make them want to publish in electronic journals. These discussions have formed the basis for building the formal project Evaluation Plan, which will be ready in August.

3.6 Journal Clusters

Studies on electronic journals indicate that critical mass is an essential ingredient for success: there must be a sufficient number of journals of sufficient interest to motivate the reader to use them. The Project Proposal contained a provisional list of journals submitted by the publishers. In January/February the publishers were asked to review these lists and develop them into clusters which in their view would achieve critical mass. The following subject areas for journal clusters were agreed:

For each cluster a publisher with editorial expertise in that subject area agreed to act as a coordinator for decisions about the cluster. The lists were reviewed with respect to scope, depth, and quality, and a list of criteria was developed for including titles.

In March the rollout of clusters was planned. Again a list of criteria was developed. Key considerations were: the availability of electronic files, multimedia potential, complexity of the text, and volume of articles/graphics. Communication and Cultural Studies (CCS) was selected as the first cluster to be launched, and Molecular Genetics and Proteins (MGP) the second.

Based on the above analyses and the assessment of fit with the user sites, we may not proceed with the cluster on Computer Graphics.

3.7 Call for Software Vendor Participation

In late February a Call for Software Vendor Participation was publicised, and software developers and vendors were invited to submit proposals. The Call for Software was sent to some 40 vendors of application software and tools. Several proposals were received that merited serious consideration, and follow-up meetings were held with the various vendors. Criteria were developed for evaluating the proposals, and the project reached agreement with Dataware (NetAnswer) and Mitsubishi in May. Further license agreements are planned. To date some “freeware” has been used in the prototype application (Isite), but this is likely to be replaced with the above software which offers greater functionality.

3.8 Application Functionality

In February the publishers were asked to profile the functionality they would like to see in the SuperJournal application. This included the core functionality like searching, browsing, and hypertext linking, and the multimedia functionality that might be suited to each subject area. This preliminary list was included in the Call for Software Vendor Participation. Functionality was further fleshed out in a special meeting of the Planning Committee in March. The end result was a list of features we felt was essential and/or desirable, and served as a basis for discussion with the software vendors. Since then we have identified those aspects of functionality that are likely to be most important, the “USPs”.

Functionality was explored in greater depth at a small focus group involving representative publishers (8 July). Issues covered included user access and registration, opening screens, options and choices available to the user, and search functionality generally. Manchester Computing have enhanced the prototype since the focus group, and a revised version was shown to publishers with journals in the CCS cluster (25 July). The next task is to agree the functionality that we feel is essential for launch, and the rollout of features thereafter.

3.9 Library Visits

In April and May the Project Manager and a researcher from HUSAT visited each of the proposed library sites to:

The librarians were given a list of the questions to be asked in advance, and the same list was used for each library. They were asked to do some follow-up tasks, eg check holdings of the journals, and fill in a technical questionnaire to profile their local technical infrastructure. Finally they were asked to further explore the fit of the journals with research interests at the university, contact relevant department to assess interest in the project, and (especially for the first cluster) identify readers willing to participate in the project evaluation.

A report was prepared profiling the user sites and a list of criteria developed for evaluating them in the context of project research. Key criteria were:

and the universities that met these criteria were:

Librarians are now being contacted to confirm their participation, and a meeting will shortly be held to plan launch of the first cluster and implementation at each site.

3.10 File Formats, Production, and Delivery

In February each publisher indicated what types of files they currently produce or intend to produce. Most publishers could supply SGML header files, and PDF or PostScript files; many planned to produce SGML full text, but not until 1997. In March they supplied sample files, and specifications to document how they implemented PDF and SGML. The sample files were analysed by an external consultant (Alden Press) who assessed the degree of variation among the various SGML DTDs used by the publishers, and the degree of harmonisation which might be reasonably achieved. They also made recommendations on what file formats the project should accept, how these formats should be specified, and options for file delivery to Manchester Computing.

Some time was taken to assess the complexity of converting files from one DTD to another, where conversion should be done, and the tools and skills needed to do it. Manchester Computing developed a “generic” header DTD for SuperJournal which contained elements from all the various publisher DTDs, a SuperJournal header DTD which defined the data elements we would extract for the journal application, and a method for conversion from publisher DTD to the SuperJournal DTDs. This was a significant step for the project. The Project Proposal had assumed that either a DTD standard would emerge that publishers would adopt, or we could convince them to use one standard for the project. Since the Proposal was written, many publishers have adopted SGML for headers, but each has developed its own DTD and has no wish to change. As the production process developed by Manchester Computing allows each publisher to continue to use its own DTD, it reflects a “real world” view and will enable the project to more fully assess scalability issues within the production process. It also places the project, and Manchester Computing in particular, to play a key role in harmonising standards for DTDs.

In May the publishers held a technical meeting with Manchester Computing and Alden to discuss Manchester Computing's overall approach for production and file handling, agree file formats for submission and outline the content for the Production Specification. Conclusions of the meeting were that the publishers would submit SGML heads and PDF pages in the first phase of the project, and submit files by FTP if tests proved successful. We plan to move to full text SGML at a pace set by the publishers' ability to produce it. An ongoing issue is how to handle journals where the publisher does not produce PDF or SGML files.

Subsequently Manchester Computing planned and implemented a very successful test of FTP transfer. It is likely that many publishers can deliver their files in this way on a production basis. Starting in May the publishers also started sending more files, so that MC could test the production processes with files in volume. Now that publishers will be sending files on a regular basis, the project is about to appoint a part-time Production Coordinator, ideally based at one of the publishers, to plan production scheduling and chase files against the schedule. If media conversion is required for specific titles, this can be outsourced (eg Alden).

3.11 Technical Development

The technical development work for SuperJournal has been done by Manchester Computing and is in three main areas:

The team at Manchester is small (one Technical Project Manager, one Data Handler, and one Developer) and the timescale for building a working system is short (6 months), so the following principles have been used in the development work:

The development environment is an object-oriented database management system (ODB-II from Fujitsu) on the Cray CS6400 at Manchester Computing. The input process involves logging in the files, parsing the SGML header files against the “generic” DTD template, and extracting the required data elements according to the SuperJournal DTD. Staff at MC have developed the DTDs and the conversion programs.

The output from processing of SGML files is used for data loading. An object base has been created in ODB-II to store the SGML elements, along with the PDF and multimedia files, and the associations between them. The output from processing the SGML files is also used to load any external databases used by the journal applications, eg BRS/SEARCH.

The user will access the journal applications using a Web browser. ODB-II manages the search and browse requests. Where text is retrieved, HTML is created on the fly for display. Where PDF is retrieved, Adobe Acrobat is invoked to display the file. The same approach is to be used for multimedia, a request for the content will be passed to ODB-II which will locate the file and invoke the appropriate application for viewing.

As of July the input, load, and retrieval processes have been created, a demonstration user application has been designed using Isite and BRS/SEARCH as search engines. The demonstration contains a few files and has been shown to publishers at two meetings. We are currently assessing various aspects of the application, its functionality and expandability, to plan how it can be enhanced in the short term, and upgraded and expanded in the long term, in conjunction with the evaluation research.

3.12 Access

Our starting point was the library visits: all the libraries wanted to know how the project would enable access for users, both within the library and from their departments. They expressed requirements from their point of view: access should be easy for users, it shouldn't involve the library in a lot of administration (recording individual passwords for users), and ideally one “username” for the whole university.

The staff at Manchester Computing looked at the options from a technical point of view, and these were considered in light of the library requirements and the usage data needed for the evaluation research. In order to answer the research questions, we need to identify individual users, both to measure their use, and to contact them so we could understand their use. A final list of requirements was drawn up, and MC developed a system for user registration which we feel meets everyone's requirements. This is currently being tested.

3.13 Letter of Understanding

An agreement has been drawn up by a solicitor formalising the roles and responsibilities of the three project partners: the publishers, University of Manchester, and Loughborough University. Though we have called it a “letter of understanding”, it is a single agreement signed by all three parties, not a letter from one to the others. A draft has been reviewed, and the final agreement should be ready for signing shortly.

In order to sign an agreement with the project partners, the publishers had to form a legal entity: SuperJournal Ltd, a company limited by guarantee and formed for the duration of the project. It took some time to form the company, and the letter of understanding could not be signed until the company was formed.

The universities that will participate as user sites will sign individual letters of commitment confirming their participation in the project and acknowledging their responsibilities with respect to it.

3.14 Self-Evaluation

As noted in Section 1, the self-evaluation needed for a knowledge-creation project is different than that needed for a system-creation project. For SuperJournal, the Evaluation Plan details the data that will be systematically gathered and analysed as a result of delivering electronic journals to user sites. The self-evaluation plan concerns the manner in which the project will use this and other data to review what is being learned, to test its validity and relevance with stakeholders, and to influence the future direction of the project.

This will involve meetings and focus groups with stakeholders, both within and outside the project, eg publishers, libraries, software vendors, and project sponsors, to “test” the knowledge as it is learned.

It will also involve analysing what are the most effective ways to gain knowledge, how effective we have been, and how we can improve our methods. We will need to be able to track back to key decisions, and understand why and how they were made. The project has therefore set up a “decision log”, and will regularly assess the outcome of these decisions.

The process of self-evaluation for SuperJournal is more fully described in a Self-Evaluation Plan, a companion document to the Evaluation Plan.

4. Outputs and Deliverables

Section 3 indicates what we have done, how, and why. This section lists some of the tangible outputs to date from these activities. Formal project deliverables are marked with (*).

4.1 Planning and Project Management Deliverables

4.2 Publicity Deliverables

4.3 Evaluation Deliverables

4.4 Development Deliverables

5. Measuring Success

The project team feels our efforts to date have been very successful, and we would note in particular the following achievements:

6. Outcomes and Opportunities