AVETRA’s journal ranking exercise: The process

During 2007 AVETRA decided to undertake an exercise to evaluate the quality of, and thereby rank, the scholarly journals in the VET and adult education disciplines. Many other discipline areas in Australia were undertaking a similar exercise, prompted by the government’s research quality framework (RQF) exercise. Although the RQF has been abolished by the new government, there will be another exercise known as the ERA, in which hard data like journal rankings may be even more important.

AVETRA decided on two basic principles:

* There would be a multifaceted approach to the exercise – it would not rely on a single data source.
* Only those journals which were used by or known to AVETRA members and other VET/adult education academics would be evaluated.

The working party consisted of Erica Smith, Michele Simons, Ruth Wallace, Ros Brennan Kemmis, and Lauri Grace. John Rosser provided valuable research and clerical assistance. We believe our process to be among the most thorough of any discipline in Australia, as many others relied only on people’s general opinions of journals rather than a more exhaustive quality inspection.

This ranking is the official AVETRA ranking and may be quoted by academics and others when wishing to establish the credibility of a journal. Nevertheless we need to emphasise that the criteria were scholarly in nature and it is well recognised that journal have different audiences. Therefore it is not intended to imply that a ‘lowly ranked’ journal is inferior in any way. It may simply be designed for other purposes instead of, or as well as, scholarly discussion.

The process had two phases:

Phase 1: A list was drawn up of around 30 journals in the adult education/VET area. This list was primarily based on a list drawn up by the SORTI research centre at Newcastle University as part of a larger exercise of ranking Education journals. A survey was sent to approximately 100 members of AVETRA (only academics were invited to respond) and to other VET/adult education academics via their Deans. There were 17 responses. People were asked to provide their opinions of journals in which they had published or with which they were familiar, so the numbers of responses for some journals was quite low. People were also asked to nominate additional journals to add to the list.

Phase 2: We contacted all of the journal editors on the amended list and invited them to send multiple copies of one edition of the journal (we asked for an edition from early 2006 to maintain some sort of parity). Not all editors responded to our request, despite reminders. We decided that we could only evaluate and rank those journals for which we were able to inspect copies, so 22 journals were evaluated in this way. A panel of three senior external reviewers covering a range of sectors within the VET and adult education discipline as well as two of the working party assessed all of the journals, proving ratings and comments on different aspects of the quality of individual articles as well as an overall judgment on the journal. Only journals which had peer-reviewing processes were included in the ranking.

A ranking table was drawn up based on the results of these ratings, and the survey (Phase 1) results were used to help decide where to draw the line between the different tiers. Each journal editor was sent a full report including the qualitative comments made about his or her journals.

More detail about the process is obtainable from Professor Erica Smith, the Convenor of the Working Party, at e.smith@ballarat.edu.au or 03-5327 9665.