SuperJournal Evaluation Plan

Home | Search | Demo | News | Feedback | Members Only

1. Overview

The purpose of the SuperJournal project is to identify the features and functionality of electronic journals that have value to the readers and authors of scientific journals in Universities thereby assisting in the design/creation of successful electronic journals in the future. There are many factors which might contribute to this value, for example, in the delivery `I can access it from my desk'; in the functionality `I can search through the full-text and then view the protein structures in 3-D' or in the content `All the journals I want to look at in my discipline are here'.

On the obverse side there are many known barriers to the use of electronic journals. Some are the reverse of the benefits; e.g. not enough material relevant to me available, not enough `added value' features to distinguish it from paper, etc. Some are additional issues. e.g. difficult to read full text on a screen, takes too long to download, etc.

In the SuperJournal project we shall aim to deliver as full a range of valued properties as possible in each journal cluster and to minimise known barriers. We will systematically vary some features of electronic journal delivery to test what is required, for example, in different subject domains. A number of user sites have been selected in order to see if the activities that the reader/author has to do is affected by the way that the sites choose to provide the information. At the functionality level, the project will provide a range of search engines and other functionality, and do so in both science and the social sciences to see if there is a difference.

At the end of the project we aim to be able to tell others what key features should be provided in an electronic journal and how they should be delivered. It is unlikely that, at any stage, we will be able to deliver all that is of value or remove all the barriers to use. The aim of the project is therefore to offer a sufficient critical mass in each domain to make apparent to authors and readers what is possible and to explore with them what would characterise the most valued service. In this way we aim to create the knowledge which interested parties need in order to plan their future contributions to electronic journals; for publishers to identify the USPs (unique selling points) for their future journals and for libraries to establish the rôle of electronic journals in the delivery of scientific material within their institutions.

Within the SuperJournal project there will be a number of phases when different clusters of journals will be delivered to different user groups within Universities and different groups of authors will be involved in the development of electronic journal papers. The successive delivery of different journal clusters provides an opportunity to change the journal content and means of delivery to explore what factors facilitate usage.

Within this framework the evaluation strategy may be characterised as longitudinal and interpretive action research:-

• action research because data collected from the initial delivery will be used to refine the subsequent deliveries;
• longitudinal because data on authors and readers will be collected through the life of the project in order to reveal changing patterns of usage and to link them with changing elements of delivery;
• interpretive because evaluation will not be restricted to behavioural data but will ask all stakeholders to reflect on this data in order to understand it and explore its implications for their perspective on electronic journals.

2. The Research Questions

The evaluation process will serve to provide progressively fuller answers to the following eight questions which relate to the positive and negative consequences of electronic journals for four key stakeholders.

The Authors Authors need to perceive value in delivering electronic versions of their papers. Therefore:

• Question 1: What factors will facilitate the provision of value added electronic articles by authors?
• Question 2: What barriers or inhibitors have to be overcome to make it possible for authors to create electronic versions of their papers?

The Readers Readers who must see additional value in electronic journals if they are to use them instead of/as well as paper based versions.

• Question 3: What factors in the content of an electronic journal, the system within which it is provided and in the local user environment, facilitate usage by readers and the judgement of added value?
• Question 4: What barriers or inhibitors in the content of the electronic journal, the system within which it is provided and in the local user environment, will inhibit and prevent the experience of added value?

The Publishers As the stakeholders who have traditionally provided the link between authors and readers, the publishers must see commercial benefit in what authors and readers value in the provision of electronic journals.

• Question 5: Of the facilitators readers and authors seek, what constitute Unique Selling Points (USPs) to publishers which they could exploit in future journals?
• Question 6: Of the factors which readers and authors seek, what constitute threats, difficulties, risks or challenges to publishers which may inhibit their further development of electronic publishing? How will they overcome reader and author barriers?

The Libraries As the traditional local mediator of journals provision to readers, the Libraries are a significant stakeholder in the widespread adoption of electronic journals.

• Question 7: Of the facilitators and inhibitors which are judged important by readers and authors, what constitute actions the Libraries could take to enhance their rôle and to facilitate the usage of electronic journals?
• Question 8: Of the facilitators and inhibitors which are judged important by readers and authors, what constitute threats, difficulties, risks or challenges to libraries which may inhibit their facilitation of electronic journal usage?

3. Methods

3.1 Modelling the Facilitators and Inhibitors

Each of these questions relates to the facilitators and inhibitors of electronic journal usage and a fundamental feature of the evaluation strategy is that it will be driven by a model of the factors that encourage/discourage usage of electronic journals. An outline framework for the facilitator/inhibitor model is given in table 1. It contains illustrative factors but is not a comprehensive listing.

Table 1
An Initial Framework for a Facilitator/Inhibitor Model with Illustrative Content

The model will be developed and used as follows:-

1. The model will be developed initially by examining past research and obtaining the input of knowledgeable stakeholders, e.g. publishers. The factors which encourage use of electronic journals will be incorporated, e.g. the added value features in the content of journals, the way they can be accessed, browsed, etc. It will also list the inhibitors to usage which may be the lack of value added features or other factors such as cost, usability, etc.
2. The delivery of each cluster of journals will seek to maximise the value added features and minimise the barriers to usage. Inevitably the provision of value added features will not fulfil all the criteria predicted to provide maximum potential. Existing research data and baseline data from the specific reader/author community in each institution and journal will be used in conjunction with the model to make predictions about the kinds of usage patterns to be expected.
3. The evaluation methods will be directed towards the predictions made by the model. The data collected will be of two kinds. Behavioural data will be collected by reader usage statistics and by studying author behaviour. This data will be used to explore with readers and authors the rationale for their behaviour and the `cost-benefits' they perceive in electronic journals, i.e. the value added features that matter to them and the barriers to authoring and using electronic journals that will affect their behaviour.
4. After each cluster of journals has been available for three months a report will be prepared. It will specify what was delivered, how well it corresponded to the `ideal' model, what usage occurred and, in particular, what cost-benefits were operative for relevant authors and users. Each report will review the model, add other variables as appropriate and give weightings to what emerge as dominant variables. The findings associated with each cluster following initial delivery will provide the basis for a review 18 months after delivery. By this stage other value added features will have been introduced, further readers and authors will have been involved and usage patterns will have matured. A second report on each cluster will provide a more considered statement of the factors facilitating and inhibiting usage for each subject.

As a result of reviewing existing literature on the usage of electronic journals (SuperJournal report on The Usage of Electronic Journals) and of focus group discussions with publishers, a provisional list of value added features and functionality has been drawn up. It is being used to prepare the delivery system for the first cluster of journals.

3.2 From Usage Modelling to Stakeholder Interpretation

While the evaluation strategy seeks to establish the facilitators and inhibitors for direct users of electronic publications, i.e. authors and readers, the knowledge that is generated will be of relevance to the range of other stakeholders who are part of the delivery chain for electronic publications, i.e. publishers, libraries, software vendors, developers, editors, etc. The data generated from journal cluster delivery will help these stakeholders develop a fuller understanding of the issues as viewed from their perspective. The aim of the project will be to systematically present this data to stakeholders and help them review the implications through the following process:

1. Stakeholders will receive the report for each usage period. They will be asked to assess this data in a cost-benefit framework, i.e. what value added features emerge as important which could be further exploited (by publishers, etc.) and what barriers need removing, by whom and at what cost.
2. Stakeholders will be encouraged to look forward at actions that could be taken to improve the delivery of electronic journals. This may involve constructing and testing scenarios based upon the maturing `ideal' model of electronic journal delivery.
3. Evaluation reports for each usage period will be reviewed by the consortium planning group and others. They will be used to review what worked and did not work in the planning and delivery of the previous cluster and to change these parameters in the delivery of the next cluster.

4. The Process of Evlauation

In this section the sequence of evaluation activities throughout the project will be described. The process is primarily a formative evaluation in that each stage of evaluation influences the next delivery but it is ultimately summative, providing a statement of our acquired knowledge on what is necessary to deliver electronic journals of value to different communities of authors and readers.

4.1 The Cluster Orientation

SuperJournal will develop by successively offering clusters of journals of interest to different communities of authors and readers. The evaluation process will have a cluster orientation; initially attempting to model the value added features to be provided in each cluster and to collect baseline data about the community of authors and readers who will be able to access the electronic journals. After delivery usage data about each cluster will be collected and followed up and a report prepared on the facilitators/inhibitors experienced by that community and the value added features of particular relevance to them. This process will be repeated for each cluster.

4.2 Base Line Studies

4.2.1 Library Visits

The participating libraries will be visited and their librarians interviewed. The aim of these visits is to learn more about the libraries' collections, organisation, management, resources and current involvement in electronic library activities. A particular interest will be the match between the libraries' holdings, in terms of the SJ journal clusters, and the degree of reader interest in those titles. At the end of this exercise the SJ team should be able to state which libraries have the best likelihood of providing access to readers of the SJ titles.

It is hoped that at least three libraries will have communities of users interested in the each journal cluster to be made available.

4.2.2 Base Line Assessments of Readers

When the initial journal cluster libraries have been identified, likely readers will be contacted. These will be members of academic staff and post-graduate students. The objectives of this initial exercise will be to:

• generate a picture of the level and pattern of users' journal reading habits (how often do they use journals, why do they refer to them, where do they read them, how do they read them)
• assess readers' current use of electronic resources and their expectations about (and enthusiasm for) electronic journals
• introduce them to the SJ project and seek their active participation in its evaluation.

Readers will be identified by way of subject specialist library staff, departmental library representatives, inter-library loan request records and self report questionnaires attached to the current issues of the SJ titles.

While E-mail questionnaires will be used to gain the above information emphasis will be placed on direct contact (given the low response rates that are widely reported for E-mail based methods). Smaller numbers of targeted users will be invited to join focus groups.

4.2.3 Base Line Assessments of Authors

Author participation in the development of electronic journals is critical to their success. Prior to the release of each journal cluster, UK university based authors will be identified from backcopies of each of the print-based journals. Questionnaire studies will be undertaken to establish the attitudes of these authors towards authoring electronic journal papers. Authors at institutions which are particularly well represented in terms of publications will be invited to attend locally held focus groups in order to confirm the results of the questionnaires.

4.2.4 Base Line Assessments of Libraries

The base line assessments of libraries will comprise interviews with the librarians undertaken during the initial visits to the participating libraries. See section 3.2.1.

4.2.5 Base Line Assessments of Publishers

A postal questionnaire will be administered to the journal publishers which will record publishers expectations concerning the impact of multi-media and the submission of electronic manuscripts by authors on the publishing process.

4.2.6 Interface Evaluations

Prototype SJ interfaces will be evaluated initially using expert critiques and user walk-throughs by staff with a wide experience of interface design issues.

Interfaces that are made available to library users will be evaluated via self report questionnaires, structured interviews and group evaluations.

4.2.7 SJ Implementation

It is assumed that the participating libraries will take responsibility for implementing SJ in accordance with their usual procedure. The SJ team will provide support as required and encourage maximum publicity with respect to potential readers. This may involve sending out specific publicity material via E-mail and internal mail and invitations to on-line demonstrations. The evaluation team will not have a major role in the promotion of the system given their role as evaluators but will record the form taken by the implementation process in each institution. A key aspect of the implementation will be the confirmation of potential users.

5. Conclusions and Deliverables

The overall evaluation programme for each journal cluster is summarised in table 2. The timescales all derive from the dates when clusters of journals are made available. Clusters which are delivered early in the project can be finally evaluated after 18 months. Clusters which are delivered in the final 18 months will be reviewed 3 months from the end of the project.

At the end of the project the evaluation will have yielded considered assessments of the provision of five journal clusters plus interpretive commentaries by relevant stakeholders. We would have a view of the value added factors appropriate in subject domains and the barriers relevant in each case. This will provide the basis for a final summative evaluation report which will present an overall model of the factors influencing electronic journals with a contingency framework which explains the variations found for different subject domains, institutions, etc. The report will also include an analysis of the future of electronic journals from the perspectives of different stakeholders, i.e. readers, authors, publishers, librarians, etc.

August 1996