2004-02-06
Evaluators wanted? - Some reflections on the National Agency's Evaluations Department
The panel would first of all like to extend its praise to the Evaluations Department for having itself initiated an evaluation that can be both time-consuming and revealing. Naturally, the National Agency will improve its own standing in being able to demonstrate that it has first-hand experience of the evaluation model in current use.
Not only were we received with great openness during our visit but the panel has also been provided with a great deal of assistance in other respects and the members of the staff we spoke to have had many interesting and important points of view about their own work and the Evaluation Department´s activities. A summary is presented below of our observations and reflections in connection with the self-evaluation and our site visit. We focus to begin with on what we consider to work well. We then go on to raise issues and make suggestions that we consider could further enhance the quality of the work of the department.
The Evaluations Department appears to be a an enjoyable place to work This becomes clear in the self-evaluation, and the interviews conducted by the panel of assessors during its visit merely confirmed this state of affairs. There is considered to exist a good mix of various skills, which helps to make the work stimulating, there is great scope for individuality, excellent opportunities for the development of personal potential and the staff enjoy working with each other. This, in the panel´s opinion, says a great deal about the quality of the Evaluations Department and about the National Agency.
The work of the Evaluations Department is thorough and professional and commands respect both within Sweden and internationally All members of the staff are very keen to ensure that their work is of high standard. Their task is both comprehensive and complicated. It is discharged with great expertise and the department is very productive. Its work is recognised as thorough and professional both in Sweden and internationally.
The department strives consciously to assure the quality of its projects and project administration
Our impression was that there is great awareness of the importance of good project administration - for the work of the panels of assessors, the quality of their evaluations and the sectoral legitimacy of the National Agency. The Evaluations Department also devotes a great deal of concern both to the selection of suitable panels and to introducing and preparing them for their tasks.
In other words, we gain a very positive impression of the capacity of the department to implement its evaluations and perform the tasks entrusted to it. Nevertheless, there are always some operational aspects that can be improved. We consider that our task requires us to make proposals concerning factors that could further augment the quality of what the department does. We would like to raise a few points that the department - administrators and members of its staff - should reflect on.
The process of developing evaluation models and also evaluation issues in general
In the department there is great agreement about how its task should be interpreted and that the evaluation model currently used is “correct". The impression we gained was that this model is considered to enjoy international endorsement and that it has gained the acceptance of the higher education sector. With one single evaluation model, however, there is a risk of uniformity. Higher education comprises many different programmes and subjects, with different aims, content and organisation. It is hardly reasonable that one and the same evaluation programme will be most suitable on every occasion. The question of how the next round of evaluations should be implemented and what the “model" should look like after 2006 was raised in the self-evaluation, but in effect reflections in the department hardly go further than that.
Our impression is that the department as a whole is uncertain whether its task involves any discussion of issues of this kind. This is expected to take place among the Agency´s senior administrators and in the Advisory Board linked to the Agency. We have no difficulty in understanding this approach in view of the staff´s heavy workload and their focus on the implementation of the various evaluation projects. Nevertheless, the question we should like to pose is how one can know what model to use in 2007, if there has been neither discussion nor any test of at least a few alternatives. This is something that we believe the staff of the department should reflect on jointly.
More explicit feedback to members of the staff
The Evaluations Department has a very flat organisation. If we disregard its administration, the members of its staff are either evaluators or Heads of Department. In other words there are no promotion routes. The National Agency is not unique in having such a flat organisation, as it is becoming increasingly common in agencies whose task is to monitor. Some members of the department´s staff point out that what is needed is not “new titles" but more direct response to the work that they do.
The evaluation staff work hard and spend a lot of time travelling, so that they sometimes feel it is difficult to keep in touch with day-to-day developments in the department. We believe that in circumstances like these it is even more important to provide individuals with information and feed-back. We gained the impression that there is a lack of both general discussion about what constitutes a good or less than adequate report and also explicit feed-back to individuals, which does not always have to be directly linked to the quality of a report. More explicit links between individual performance and the criteria for salary increments were also missing.
Define the internal support
On the whole, members of the department´s staff are offered excellent conditions in which to carry out their work. Excellent facilities are provided in the form of technical aids, a library and an augmented project management team. Opinions were divided, however, about how well the administrative support functioned. Some of the staff experience a heavy administrative burden. This may to some extent be self-inflicted, as some project managers like to have “control" over every element in “their projects". It may also be due to the lack of any clear definition of what the role of project manager entails.
One thing that was felt by everybody to be a pointless routine, as nobody takes it seriously, is accounting for time. The panel of assessors considers that a decision must be made about whether accounting for time is to be retained or not. If it is to be retained, and it is important from a planning and accounting perspective, then it must be clearly indicated how time is to be accounted for and how this information is going to be used and fed back into the system.
A long-term strategy for the enhancement of the expertise of the entire Evaluations Department
A great deal of attention is paid to the desires of individual members of the staff to enhance their expertise by attending conferences and seminars. Declared policy is that it is mainly up to the individuals to take the initiative and responsibility for the development of their own qualifications. On the other hand, there is no policy or strategy about ways in which the department as a whole can benefit from the knowledge and experience of individual members of its staff.
Presentation of project reports at departmental meetings is important and a precondition for any collective learning in the department. But this is not enough. The panel of assessors believes that one possible reason for the lack of any joint policy in this area is that the department is constantly working against the clock. Another possible reason may be the difficulty in defining the skills that the department needs to develop in the long term. Project management training and understanding of the system of higher education, presumably acquired prior to appointment, are naturally among the qualifications required of the members of its staff. It is less clear, however, what else is needed. As no subject expertise is required of a project manager, and indeed such a requirement would be unviable in view of all the subjects and programmes that have to be evaluated, the question that then arises is what skills need to be developed. Are they skills in the field of evaluation? If, as noted above, it is considered to the task of the Agency´s policy makers to discuss the evaluation models to be adopted, what skills remain in which competence can be enhanced?
The panel of assessors considers that further discussion is required about the skills needed by project managers in the Evaluations Department. This is not only important to enable individual members of its staff to feel that support is available for their development in the department but also to provide the department with a shared frame of reference that will enable joint discussion and collective learning.
A strategy to enable the integration of the entire National Agency for Higher Education in the process of evaluating subjects and programmes A number of measures have been adopted to ensure greater coordination between the National Agency´s various departments. Nevertheless, from the information presented to the panel of assessors it would still appear as if the Agency´s administration should take even more responsibility for increasing inter-departmental exchange. Here, we are thinking primarily of cooperation between the Department for Planning and Research and the Evaluations Department.In conversations with the staff of the Evaluations Department it was pointed out, for example, that better use could be made of the Monday Seminars for exchange of information between the departments. Increased participation would also be welcome in the follow-up seminars arranged by the Evaluations Department for representatives of the subjects and programmes that have been evaluated. We believe that there are several explanations for the dearth of exchange of experience between the Evaluations Department and the Department for Planning and Research. One is that the staff of the Evaluations Department, as even the Department for Planning and Research acknowledge, have the most demanding workloads in the Agency. Another is that the evaluation projects often extend over more than one year and it is difficult to plan job-rotation between departments for such a long period. Further explanation may also be found in the uncertainty about the contributions that could be made by staff from other departments to an evaluation in any specific area. As we arrived at the conclusion that no extensive evaluation expertise was required but rather project management skills, adequate knowledge of the higher education system and good analytical capacity, some members of the staff of the Department for Planning and Research should be well equipped to take part in evaluations.
We feel that a strategy is needed to further facilitate cooperation between departments so that more members of the staff of the National Agency are able to take part in evaluations. This would also increase the contacts of other members of the Agency´s staff with the higher education institutions and perhaps also reduce the workload for the staff of the Evaluations Department. For example, staff from Planning and Research could participate in the evaluations of subjects and programmes and Evaluations Department staff could undertake Planning and Research tasks.
The organisation and content of the reports
We have of course not been able to read all the evaluation reports but even so there are good reasons for reflecting on their target groups, their organisation and content. In our conversations with the staff of the National Agency, we have been told that these reports are the assessors´ reports. It is the experts in the panels of assessors that make the evaluations. The question that we pose is what function is fulfilled by the sections of the reports produced by the Agency itself, apart from those that describe decisions on the right to award degrees, which is one of the specific functions of the Agency.
We can generalise to some extent and describe the reports as follows. An evaluation begins with a summary. The Agency´s own reflections are, by and large, a summary of the entire report in which agreement is expressed with the opinions of the panel of assessors. In addition, the panel itself often includes a brief summary in its own section of the report, and also an outline of its assessment and recommendations or the like. This means that four types of summary can be found in one and the same report, so that they become repetitive and it is unclear to the reader who is saying what.
The question that arises is whether in future there should be more explicit demarcation of the National Agency´s contribution. If the Agency has nothing to add to the conclusions reached by the panel of assessors, it should consider whether anything more is needed than its decisions. Alternatively, the Agency could elaborate on its own points of view in relation to those of the panel. We consider it important to continue discussions in the National Agency about what opinions it either must or should express, and also in what areas.
Consideration should also be given to whether the panel of assessors necessarily has to be totally unanimous in its assessments. It could well be the case that the description in a report of different points of view, or even some form of “disagreement" insofar as there is any, could be interesting and useful for those who are being evaluated.
The Evaluation Department´s self-evaluation
In our interviews, many of the staff were very self-critical of the excessive descriptiveness of the department´s self-evaluation. The analysis “simply vanished" and the staff found that it was difficult to complete the analytical section of the self-evaluation. What was missing was mainly assessment of what they were good at, what they believed they could achieve, etc. There is discussion of strengths in the form of structures and process but not of what they result in. There is also a lack of any more penetrating analysis of what could be improved in the process with the assessors and what can be learnt from experience of the work that is being done. Analysing oneself is difficult according to the departmental administration.
We expected to hear during our interviews some reflection on attitudes to self-evaluation as a method and how the Evaluation Department´s own experience of self-evaluation could be transferred to the National Agency evaluation model. The reflections made by the members of the staff themselves is that there are shortcomings in their own self-evaluation - of the same kind as those found in the self-evaluations in the higher education sector. This should give rise to a number of questions. How well does the evaluation model function? What should we bear in mind when reading the self-evaluations and making site visits? What can we do in the future to provide support for the analytical elements in self-evaluations? In other words, an important factor in the Evaluation Department´s reflections about its model should be consideration of the role of self-evaluation and how it can be developed to become more analytical.
The National Agency´s model is not very suitable for an evaluation of this type
We realise that the most important aspect of this evaluation for the Evaluations Department has been to test the model on its own activities. Nevertheless, in “the spirit of evaluation" we should like to take this opportunity to express some reflections about the evaluation itself and the evaluation model. The difference between evaluating a subject or programme in higher education and what we have done is that normally panels of assessors are appointed that represent subject expertise in the area in question. In this panel we were not experts on the operations of the Evaluations Department. The approach adopted does not provide the depth required for a thorough assessment of the process in the department and what conditions need to be fulfilled to ensure the effectiveness of that process.
If we had been given the task of evaluating the Evaluation Department´s doings without having to use a specified model, we would have monitored its processes more closely. We would have liked to follow one evaluation from beginning to end. This would have involved participating in several site visits to enable assessment of how this particular aspect of the process functions. Similarly, it would have been necessary to participate in a number of the initial meetings with the panels of assessors to see how their frame of reference is created, as this is decisive for the entire evaluation.
Summary
Our overall assessment is that the staff of the Evaluations Department carry out their duties professionally and that the department is a pleasant place to work in. Nevertheless, there are critical observations about the Evaluations Department and improvements are always possible. Some of these improvements involve the National Agency as a whole. The proposal to intensify cooperation between the Evaluations Department and the Department for Planning and Research, for instance, clarification of the National Agency´s contribution to the evaluation reports and concrete work on various evaluation models are examples of the measures that not only concern the Evaluations Department but the Agency in its entirety.
Print
Phone: 08-563 085 00 Fax: 08-563 085 50 Email: hsv@hsv.se
