Assessment and Policy Development in Sustainability in Higher Education with AISHE

The Declaration, Volume 5, Number 2: May 2002  [Research]

This article includes excerpts from an expanded article that will appear in a future issue of the International Journal of Sustainability in Higher Education.

By Niko Roorda

The Dutch approach to the development of Sustainability in Higher Education (which from now will abbreviated to “SHE”, for short) has so far been successful. As in many countries, there are numerous initiatives in place at individual universities. But there is also a national committee, the “Committee on Sustainability in Higher Education” (CDHO). It started in 1998 as a rather informal collection of individual enthusiasts working in various universities, who sought a way to strengthen and help each other in their pioneering attempts to integrate sustainability in the educational programmes. In fact, it was students who took the initiative and formed the CDHO.

Between 1998 and now, the CDHO has taken the lead in the development of SHE in the Netherlands. The committee is financed by the Dutch Government (the Ministry of Environment). Besides representatives of the major Dutch SHE projects, it consists of representatives of the Ministries of Environment, Education, Agriculture and Economical Affairs, and two rectors of universities.

The committee functions not only as a network organisation, but has also initiated a number of its own activities. For instance, there is a national project called “disciplinary reviews sustainable development”, which has produced publications focused on implementing sustainability in individual university disciplines. Published so far are reviews on Management (Jonker and Grollers, 2001); Economics (van den Bergh and Withagen, 2001); Physics (Bras-Klapwijk, 2001); History (van Zon, 2001); Biology (van Hengstum, 2001) and Mathematics (Alberts, 2001). Other disciplines will follow. Plans exist to have them translated into English, in co-operation with the Swedish MINT group (the Swedish equivalent of the CDHO).

Another action of the CDHO was the formation of a working group assigned to develop a set of criteria for SHE. Soon, this working group decided that just the development of criteria was not enough: in order to operationalise these criteria, it was necessary to develop an assessment instrument now referred to as the Auditing Instrument for Sustainability in Higher Education (AISHE).

The Assessment Instrument
The European Foundation for Quality Management had developed a model for quality management, named EFQM after this organisation. As a basis, they used the Deming Quality Circle. To the parts “Plan”, “Do” and “Check” were attached a number of criteria concerning the quality management in a company. The Dutch organisation for quality management, INK, enhanced the EFQM model by attaching to each of the EFQM criteria an ordinal scale of five stages. Each stage is a verbal description of a possible state the assessed company is in with respect to this criterion (see: INK, 2000). Starting with this EFQM-INK-model, a group of Dutch universities for Vocational Education (“Hogescholen”) made an adapted version for Higher Education (which may be called the EFQM-HE-model, see: HBO Expert Group, 1999). It is this EFQM-HE-model which has been taken as the basis for AISHE.

AISHE : stages and criteria
A general description of the five stages which together form an ordinal scale, is shown in Table 1.

Table 1: General description of the five stages

Although the general description of these stages in AISHE matches those of the EFQM-HE-model, the criteria refer specifically to sustainability in higher education. The criteria are listed as follows:

== Plan ==
1. Vision and policy
1.1. Vision
1.2. Policy
1.3. Communication
1.4. Internal environmental management

2. Expertise
2.1. Network
2.2. Expert group
2.3. Staff development plan
2.4. Research and external services

== Do ==
3. Educational goals and methodology
3.1. Profile of the graduate
3.2. Educational methodology
3.3. Role of the teacher
3.4. Student examination

4. Education contents
4.1. Curriculum
4.2. Integrated Problem Handling
4.3. Traineeships, graduation
4.4. Speciality

== Check ==
5. Result assessment
5.1. Staff
5.2. Students
5.3. Professional field
5.4. Society

As an illustration, criterion 2.3 (staff development plan) is shown in detail, with all five stage descriptions, in Table 2.

Table 2: Criterion 2.3 – Staff Development Plan

During 2000 and 2001, the list of criteria was designed and discussed with many stakeholders from within and outside of education (the details of this development have been published in Roorda, 2000), and for each of the criteria the five stages were designed. In the second half of 2001, the development was completed with a series of practical tests in universities in Sweden and the Netherlands. The procedure of these tests, as well as some results of one of these, will be described below.

The Assessment Procedure
In short, the procedure for an assessment is as follows (if a minimum scenario is followed):

1. Preparation with the internal assessment leader:
• Explanation of the method
• Discussion of the procedure
• Selection of criteria and appendices to be treated
• Composition of the group of participants

2. Written information to the participants

3. Introduction with the group of participants:
• Explanation of the AISHE method
• Discussion of the procedure

4. Filling in the criteria list: by the participants individually

5. Consensus meeting with participants and consultant

6. Review with internal assessment leader

Some of these steps are explained in more detail below.

Group of participants
In small organisations (up to about 15 staff members) each staff member can participate. In larger organisations a group of 10 to 15 participants is selected. The group has to be representative of staff members and students: there should be one or more managers, a number of teachers (professors, lecturers, etc.) coming from a wide variety of disciplines, some students, and perhaps one or more members of the non-teaching staff.

Filling in the criteria list (individually)
After the model has been explained to all participants, they are asked to read the part of the AISHE book that contains the descriptions of the five stages for all criteria. While doing this, individually, they compare this to their own organisation (e.g. an education programme or a faculty of their university), and find the stage which most resembles their own situation. At the end, they write their conclusions down on a form and hand it to the assessment leader, who combines all the conclusions on one composite form.

Consensus meeting
Next, a consensus meeting takes place in which all of the participants are present. At the beginning (or earlier) the copied composite form is distributed. As before, every participant has the AISHE book, in which their own scores and annotations are written: these are essential for the meeting. All participants have an equal weight in the discussions, in the proceedings as well as in the decision-making.

The group discusses each (selected) criterion and comes to a common conclusion about the right score of the organisation. If possible, decisions are made based on consensus. If, however, for some criterion no consensus can be reached, the chair will conclude that, of all proposed scores, the lowest is chosen. This is because a higher score has only definitively been realised if all participants agree with it. In no case are decisions made by voting.

Desired situation, priorities, policy
During the discussion of the criteria, naturally a number of possible improvement points will be raised. This will enable the group to formulate – for each criterion – a desired situation. This desired situation is defined not only in the form of a stage to be reached, but a series of concrete targets and associated activities that will lead to the desired stage.

When for all 20 criteria, or for a major part of them, policy intentions are defined in this way, a large list of goals and activities will be formed on which work can be done in the coming period. The danger is that if this list is too long, some items will be ignored. It is well-known that a policy plan with more than 3 to 5 priorities has less chance of success. This is why the meeting ends with the assignation of those elements in the list of policy ideas that the group judges are most important.

The result
•A description of the present situation, in the form of a number (the stage) for each criterion plus a description for each criterion in words;
•A ditto description of the desired situation;
•A date on which this desired situation has to be reached;
•A list of first priorities that are considered to be crucial for the policy to be successful.

In the end, this package has the status of “recommendations to the management”. This set of recommendations has a good chance of being accepted and becoming a part of a concrete policy plan. This is because the management itself is represented in the group of participants (which is exactly why that is so vital!); and a representative group from the staff and the students has, if all went well, chosen the recommendations by consensus. Thus it is likely that there will be strong support for the conclusions. For an assessment in which all 20 criteria are investigated, the consensus meeting(s) can take 4 to 6 hours.

The Case of Hogeschool Himbreeg and AISHE Reliability
The instrument was tested at the Hogeschool Himbreeg, Netherlands (in fact the name “Himbreeg” is fictitious, in order to keep results anonymous), a university for vocational agricultural education. The AISHE development group was fortunate to have the opportunity to try the assessment with two different groups within the “Food Technology” study programme. Theoretically, the two groups were 100% equal, each consisting of the same number of managers, teachers and students from the same study programme. This enabled the investigators to test the equivalence between groups, an important aspect of the reliability of the method. The results, when compared, are remarkably equivalent (see Table 3).

Table 3: Hogeschool Himbreeg – Assessment of Food Technology programme

This proves that AISHE rendered (at least in this case) a very reliable result. Most of the “present” scores are identical; only 3 out of 16 scores differ. The “desired” scores show more difference, but that is no surprise since this is not a measurement but the result of a group discussion about possible future developments. Nevertheless, the total policy ambition in both groups is almost equal (16 vs. 16.5). Perhaps this value in some way reflects the organisational culture.

The resemblance between the two group results is all the more remarkable because there appeared to be a noticeable difference in the atmosphere during the consensus meetings: members of one group were rather “pro” sustainability, while some of the members of the other group showed more scepticism. Also, most of the priorities are the same in both groups. It is interesting that most of them are in the “Plan” part. This is related – as both groups explained – to the fact that the Plan-Do-Balance is not in equilibrium. According to both groups, the “Plan” part is low, compared with the “Do” part, indicating that the management and the staff of the study programme are doing quite with the education itself, but also underestimating the importance of anchoring sustainability achievements in the vision and policy.

Appreciation and Effects of the Assessment Results
At certain predetermined moments during and after the assessment, participants were asked to fill in brief questionnaires on the assessment. From the answers, it appears that the participants were enthusiastic about the way sustainability is approached with AISHE, and they believe it is a valid investigation tool. Also, the application of AISHE made it clear where the strong and the weak points of SHE in the university are, both in attempts to implement sustainability in education and in the organisational policy. A good illustration of this is the fact that most of the participants did not know that the university had already signed the Charter for Sustainable Vocational Higher Education (a Dutch charter comparable to the Copernicus Charter or the Talloires Declaration). “Communication” (criterion 1.3) was one of the items that got a high priority (which tends to be the case).

The management of the study programme are in agreement with the faculty. They too are enthusiastic about the AISHE assessment. The results form a solid starting point for the improvement and structuring of policy development for sustainability. The faculty consider this very important, since the subject of their study programme (the food sector) is particularly dependent on a sustainable future.

Unfortunately, at the time this article was written, the policy plan was not finished, so it is impossible to show that it includes a commitment to sustainability. But the management made it clear that a part of the budget certainly will be dedicated to implementing the recommendations that resulted from the assessment. The relevance of AISHE, according to staff and management of the Himbreeg Food Technology department, is reflected in the fact that shortly after the assessment a “general” EFQM-HE assessment was done by the same department.

Future Developments
With several of the universities that were assessed, it was agreed that in one to two years the assessment will be repeated, perhaps even in all of them. This will enable the project team to investigate which part of the plans resulting from AISHE appear to be successful.

In the mean time, the number of assessments will grow. Now that the AISHE instrument has been tested, evaluated and completed, a follow-up project has started (again financed by the Dutch Ministry of Environment). During this second project, the AISHE team will also be able to work as consultants, assisting universities that are working on the implementation of sustainability.

References
Alberts, G. (2001): Wiskunde en Duurzame Ontwikkeling. Netwerk Duurzaam Hoger Onderwijs en UCM/Katholieke Universiteit Nijmegen, Netherlands

Van den Bergh, J. en Withagen, C. (2001): Economie en Duurzame Ontwikkeling. Netwerk Duurzaam Hoger Onderwijs en UCM/Katholieke Universiteit Nijmegen, Netherlands

Bras-Klapwijk, M. (2001): Natuurkunde en Duurzame Ontwikkeling. Netwerk Duurzaam Hoger Onderwijs en UCM/Katholieke Universiteit Nijmegen, Netherlands

EFQM Model (1991). European Foundation for Quality Management. www.efqm.org

HBO Expert Group (1999): Method for improving the quality of higher education based on the EFQM model. 3rd version, Hanzehogeschool (representative), Groningen, Netherlands. Translation of: Expertgroep HBO (1999)

Van Hengstum, G. (2001): Biologie en Duurzame Ontwikkeling. Netwerk Duurzaam Hoger Onderwijs en UCM/Katholieke Universiteit Nijmegen, Netherlands

INK (2000): Gids voor toepassing van het INK-managementmodel. INK’s Hertogen-bosch, Netherlands

Jonker, J. en Grollers, R. (2001): Duurzame ontwikkeling in de Bedrijfskunde. Netwerk Duurzaam Hoger Onderwijs en UCM/Katholieke Universiteit Nijmegen, Netherlands

Roorda, N. (2000): Auditing Sustainability in Engineering Education with AISHE. Entree 2000 Proceedings, Belfast UK. EEE Network, Brussels.

Van Zon, H. (2001): Geschiedenis en Duurzame Ontwikkeling. Netwerk Duurzaam Hoger Onderwijs en UCM/Katholieke Universiteit Nijmegen, Netherlands

Websites
Committee for Sustainable Higher Education (CDHO): Secretariat through University of Amsterdam, tel. +31 (0)20 5256266, www.dho21.nl.

European Foundation for Quality Management (EFQM), www.efqm.org.

INK (formerly: Instituut Nederlandse Kwaliteit), www.ink.nl.

Niko Roorda, MSc, was a co-developer of a new study programme on Sustainable Technology. He was the head of this programme until 1998, when he started a project in the Brabant University of Vocational Education, called Project Cirrus, aiming at the implementation of sustainable development in more than 10 technical university programmes in the Netherlands. He started working on the development of the AISHE assessment tool in 2000. More detailed information about the AISHE method can be obtained from Mr. Roorda: nroorda@planet.nl.