[SJ Logo]SuperJournal: What Readers Really Want from Electronic Journals

Home | Search | Demo | News | Feedback | Members Only

[HRule Image]

Paper presented at ELVIRA4: Electronic Library and Visual Information Research, Milton Keynes, 6-8 May 1997

Dr David Pullinger, Macmillan Publishers Ltd, London, UK, d.pullinger@macmillan.com (Project Director, SuperJournal)

Christine Baldwin, Information Design & Management, Oxford, UK, c.baldwin@dial.pipex.com (Project Manager, SuperJournal)

Introduction

SuperJournal is a project in the eLib Programme researching the factors that will make electronic journals successful and of real value to the academic community. Many studies have explored usability: the barriers that need to be minimised so that readers will use an electronic journal. SuperJournal focuses on usefulness: the key features that will make electronic journals valuable to researchers and compelling to use. This paper describes the project and its objectives, the design of the research, the application developed, and initial results on reader expectations.

Project Objectives

The purpose of SuperJournal is to learn what features and functionality should be present in electronic journals to deliver real value to the academic community, and to explore this value from the point of view of the different stakeholders: authors, readers, publishers, and libraries:

Of particular interest is the role of multimedia: content that may be an integral part of the author's research but is not now included in printed journals or cannot be communicated in print. Examples might include 3-dimensional graphics, supporting data sets, interactive equations, video clips, or sound.

Designing the Research

The objective of the research is to answer the question: what do readers and authors really want from electronic journals? The problem is that you cannot ask readers the question directly, or if you do, they can't answer it. They need hands-on experience using electronic journals to provide a context for their views and opinions. The project has therefore adopted the following method:

The project uses an action research approach, where the results are fed back into the research itself. For example, if a particular feature is disliked, it is changed in line with comments made. If a feature is requested, it is implemented where possible. The following sections describe in more detail how the method outlined above has been implemented for SuperJournal.

Journal Clusters

Participating publishers contribute the content of established refereed journals to form the journal clusters in different subject areas:

The cluster approach is one way to address critical mass, with a typical cluster containing 10 or more journals. The cluster approach also allows us to compare the results in different subject areas.

Designing the Application

The purpose of the electronic journal application is to deliver features and functionality so that readers can tell us those they value most. The application is therefore a testbed, rather than a true electronic journal. A key design consideration is that we must be able to vary over time the features offered and the method of delivery.

We started by developing a list of value-added features that we thought would be important and wanted to test out. These value-added features are in the following areas:

In designing an application to test these features, key considerations were time and resources. As the project is three years in length, we needed an application up and running, and available to all user sites within one year. This meant that developing application software from scratch was not a viable option. Instead our approach was to find off-the-shelf software which provided the functionality we wanted, and to assemble the different components in a way that would be seamless from the user's point of view. ICL kindly licensed ODB-II, Fujitsu's relational and object-oriented database, and this was used as the development environment for “gluing” the components together.

Features and Functionality

In order to guide the development work, a rollout plan was developed for implementing the value-added features and enhancing them over time. The first release of the application, available with the first journal cluster, had basic functionality. As each successive cluster is launched, the application is upgraded with new features. By phasing each release with the launch of a new cluster, response to new features could be monitored as part of the evaluation studies. The functionality for the first cluster (CCS) is outlined below, with an indication of planned enhancements:

Hierarchical selection

Searching

Display

Other Features

Access

The following criteria were used in designing the user registration process:

The system designed works as follows. SuperJournal is available on the World Wide Web at http://www.superjournal.ac.uk/sj. Each university chooses its own username and password and method of telling users what they are, eg email, Web page. The user registers using them, fills in name/address/email details and selects a personal ID. After the initial registration, users log in using their email address and personal ID, which they can change at any time, and find out from Manchester Computing if they forget it.

Baseline Research

Eight UK universities are participating in the evaluation research and are partners in the SuperJournal project. Three universities participate in the evaluation research for each journal cluster, though all eight sites have access to the application and all content. The university sites participating in the research for the CCS cluster are University of Warwick, University of Ulster, and University of Birmingham.

The purpose of the baseline study is to:

At the time this paper was written, the baseline research for the first cluster (CCS) was nearing completion, and by the time of the conference the baseline research for the second (MGP) will also be complete. Potential users at participating sites are asked to fill in a questionnaire available on the Web. Representative users are invited to a focus group to explore their perceptions and expectations in greater depth. A typical focus group includes 4-10 users, typically academic staff, but also postgraduates. Each session might last 90 minutes, and discussion is guided by a researcher from HUSAT.

Preliminary Findings

The preliminary analysis of the baseline data for the CCS cluster suggests the following:

Current Methods

The questionnaire indicated that the readers surveyed:

The focus groups allowed us to discuss current methods and views of printed journals in greater depth. In both cases time and availability were important issues. Typically readers go to the library to find articles, make photocopies, and bring them back to their offices to read. Characteristics of printed journals valued are: a journal issue is quick and easy to browse through, you can make a photocopy, that copy is portable and readable anywhere, and it is yours to own forever. Characteristics disliked are: the article you want may not be physically available (it's stolen or borrowed from the library, or the library may not subscribe), and it's time consuming to get articles (visiting the library, interlibrary loan, photocopy queues). Demands on their time are such that they don't always have time to read the article once they get a copy.

Expectations and Needs

The questionnaire indicated that readers feel electronic journals would be useful if they are accessible, portable, and easy to use, but their expectation is that they will not be. In the focus groups the needs they outlined centred on time and availability, and there was an expectation that electronic delivery could overcome the disadvantages of print. Several facilitators were mentioned:

Barriers were mentioned as well:

The readers represented a wide range of specialist research areas, and there was a variable match to the journals offered in the CCS journal cluster. It remains to be seen whether the match will be sufficient to attract high use.

Issues and Concerns

A number of interesting issues and concerns were raised in the pre-launch focus groups. A few are mentioned here to illustrate the range of issues that might emerge from the research. One group of issues centres on readers as authors:

Other issues will be of interest to publishers and libraries, for example:

These are some preliminary results on the expectations and concerns of scholars in the area of Communication and Cultural Studies. One important question for the research is whether scholars in other disciplines will have different expectations and concerns.

Informing the Research

The baseline studies do not tell us what readers want. They inform and shape the research that will tell us what they want, and they provide a baseline of expectation to compare with actual use. At a practical level they identify areas where the project should design tests, especially for the facilitators and barriers.

Availability and Accessibility

Availability and accessibility are important issues to readers. We can test access methods: availability of the application on the Web, access from Manchester Computing via their MIDAS service on SuperJANET, whether readers feel the registration process is easy to use, and if login using their email address and a personal ID they can choose/change is convenient. We can track use and find out when and how often individuals users log in, and whether they do so from their offices or homes. And we can explore with the librarians the success of the various methods they use to inform readers about SuperJournal and encourage them to use it.

Full-Text Searching

The application provides users with a choice of search engines and different approaches to searching: free text, field specific, and natural language. The Preferences feature allows the user to select one of the search engines as a default, and to change this preference at any time. The searching that is done and the abstracts/articles retrieved are recorded. It will be interesting to see whether users opt for simple single-term searches, or exploit the capabilities of the more advanced search engines for complex searching, and spend time refining their searches to get the most relevant articles.

Alerting Service

Like availability, time is an important issue for readers. SuperJournal offers an alerting service that will change over time. Initially readers can receive tables of contents by email for journals which they select using the Preferences feature. Later they will be offered alerting at article level, based on personal profiles. Information is recorded on who uses the alerting service, how often, and for what journals. We can then explore with them whether it does save them time, keep them up to date, and what they do when they receive the alert.

Multimedia

This paper focuses on the reader research, but another whole area covered by the project is research on authors: how they perceive the benefits of electronic journals and react to the implications of features that readers want. An important aspect of the author research concerns multimedia or the “unprintable elements” that can be included in electronic journals. Authors will be invited to submit content to accompany their articles in non-traditional formats, eg film, video, sound, animations, simulations, 3D graphics, or data sets. This new content might have been generated as an integral part of the research described in the article, or might be included for illustration purposes. We will explore with readers their reactions to this new content, when it enhances their understanding of the concepts being presented, and when it is perceived as simply a “gimmick”.

These are examples of facilitators flagged by readers at an early stage, and special care can be taken to ensure that testing is done in these areas. As the research records response to these and other features, a feedback loop is invoked. Features that are used and prove to be important can be enhanced. The application, and the testing based on it, changes over time in response to what readers want.

Interpretive Review

The baseline studies inform the research at a practical level, but also at a conceptual level. They flag issues that may be important, should be tracked as the project progresses, and which are likely to benefit from discussion across the range of stakeholder groups.

Interpretive review is an important aspect of the research method. As results emerge, they will be fed back to the target audience for validation, eg results of the reader research will be fed back to readers. They will also be discussed with the wider group of stakeholders (authors, editors, libraries, publishers) to identify opportunities and issues of common interest, and to identify action that should be taken. The combination of concrete evidence of the research together with informed debate on the consequences will enable the scholarly community to design the electronic journals of the future, and deliver to readers what they really want.

Acknowledgements

SuperJournal is a collaborative research project involving many institutional partners and individuals. The main project partners are the SuperJournal Consortium of publishers, HUSAT Research Institute (Loughborough University), and Manchester Computing (University of Manchester). Universities participating as user sites are: Birmingham, Cambridge, De Montfort, LSE, Oxford, UCL, Ulster, and Warwick. The project is funded by the Higher Education Funding Councils through JISC as a project in the Electronic Libraries (eLib) Programme.

The project would like to acknowledge in particular the work of the following individuals: At Loughborough University: Professor Ken Eason, who is leading the evaluation research, and John Richardson, who performed the baseline studies described in the paper. At University of Manchester: Ross MacIntyre, Technical Project Manager, who is leading the technical development; Yamei Xu, who developed the SuperJournal application; and Ann Apps, who developed the production processes.


This web site is maintained by epub@manchester.ac.uk
Last modified: June 26, 1997