Evaluation and Research Advisory Group
Association of University Leaders for a Sustainable Future (ULSF)
A Consultation held in conjunction with
“Higher Education for Sustainability:Towards the World Summit on
Sustainable Development (Rio +10) in 2002”
The COPERNICUS Campus annual conference
Lüneburg, Germany
Introduction:
This purpose of the Evaluation and Research Advisory Group is to advance our thinking and research methodology in two areas of assessment of sustainability in higher education (SHE): a) surveys and instruments for measuring SHE; and b) case studies in SHE. In terms of assessment instruments, we are looking at existing and emerging surveys and instruments, such as the Dutch AISHE method, the U.K. sustainability reporting tool, the Campus Ecology Program’s U.S. survey of SHE, and ULSF’s Sustainability Assessment Questionnaire, in order to better understand how to define SHE. That is, by their very design and content, these instruments define how “sustainability” is currently and best reflected in university curricula, research, student activities, and operations. They define what we imagine a “sustainable” college or university would look like. With respect to case studies, we are interested in the further development and refinement of a case study methodology for exemplary institutions, that is, colleges and universities around the world that have demonstrated real commitment to pursuing sustainability in their teaching and practice.
Assessment instruments and case studies of sustainability in higher education are fundamentally linked because in order to understand and compare the efforts of different institutions (from one or more regions of the world), we believe we must first agree upon a framework for understanding how a committed university would actually function. The instruments for measuring SHE provide the framework for conducting deep and rigorous case studies.
NOTE: We acknowledge that all of the instruments listed are heavily weighted toward environmental sustainability, and that they are Northern and Western in orientation. Thus a major challenge before us is to more fully understand and reflect the economic and social dimensions of sustainability as they relate to higher education, especially as we start to apply our assessment and case study methodologies to universities of the global south. We must also continue to work with the multiple definitions and conceptions of “sustainability” for different individuals and institutions in different settings.
The goal of this Consultation of the Evaluation and Research Advisory Group in Lüneburg was to further the development of a case study methodology in light of existing assessment instruments in sustainability in higher education. ULSF is intending to hold two more consultations of the Advisory Group, one in early 2002 and the other in early 2003.
MINUTES:
Evaluation and Research Advisory Group Consultation –
Part 1
Sunday, October 7, 2001
Lüneburg, Germany
Attendees: Dina
Berzina, Heloise Buckland, Kathy Cacciola, Wynn Calder, Rick Clugston, Peter
Blaze Corcoran, Sokorro Delgado, Walter Leal Filho, Charles Hopkins, Rogier van
Mansvelt, Niko Roorda, Suzanne Savanick, Kim Walker, Arjen Wals
Chair: Peter Blaze Corcoran
General focus of the meeting: Understanding and developing an effective case study methodology for higher education institutions pursuing sustainability in higher education.
Agenda:
I. Overview of Agenda
II. Self introductions
III. ULSF goals on assessment/others’ goals on assessment
IV. Framework: conditions, indicators
V. Review of work of previous consultation
VI. Advancing the methodology (small groups)
VII. Conclusion: sharing small work, next steps
In the opening large group discussion, a general concern was raised that case studies in sustainability in higher education to date have not been very successful as a method for moving sustainability in higher education forward. (This assertion did not take into account other factors that are contributing to resistance to change in the disciplines and in higher education generally.) Given this assertion, how can case studies be improved and made more useful?
-----------------------------
The following “challenge” was used as a framework for small group discussions:
We have been asked to evaluate the current state of sustainability initiatives at 20 or so colleges and universities (located in a variety of cultures and geographies, of different sizes and types, etc.) that have claimed to be leaders in sustainability in higher education and/or have received external grants to strengthen their sustainability initiatives. Each has a stated commitment to pursue sustainability comprehensively, that is, by transforming research, teaching, outreach and operations to make sustainability a major focus. Our task is:
This should be done in one or two visits to each site, with assistance from those at each institution. We will need to report these results on an institution-by-institution basis, giving sufficient contextual data to describe the institution and its history.
The primary task for this advisory group meeting is 1. to discuss precisely how we would gather the information needed to address the four points above, i.e., who would we meet with? What questions would we ask? What institutional data would we gather? What sort of questionnaires and surveys would we design? What process would we use to ensure our conclusions were reliable and valid?
-----------------------------
First Small Group Discussion:
Based on the "challenge" put forward, the process for conducting site visits would include:
1. Meeting with a cross section of people, e.g.,
- Administrators (President/Chief of Facilities)
- Students
- Faculty
2. Exploring what is the understanding of "sustainability" at that institution
3. Working with an internal person (professor or administrator) to make introductions, such as a senior person. There is a strong need to be able to 'connect' with participants.
4. Meeting in groups and individually for interviews
5. Conducting the study in collaboration with the university:
- Making joint decisions about research tools
- Triangulating, i.e., involving different groups in same/similar questions
- Identifying key people to talk to - the enthusiasts as well as a cross section
6. Reviewing records and documentation regarding relevant projects, etc.
7. Creating a situation in which the institution is invested and assisting in the research process: i.e, university representatives would understand the role of the research/evaluation/consultant, would be realistic about how much the consultant can do for the university, and would create a process which would continue after the consultant leaves
8. Creating a framework for the study, i.e.,
- Does the consultant go in with a preconceived framework or vague ideas?
- Who decides?
9. Addressing other fundamental questions, such as:
- How can we help people create a map of where they are at, and their potential for future progress? Need a process that is not too daunting.
- How is sustainability being institutionalized?
- How do we proceed from here?
Second Small Group Discussion:
Keys are: participation; ownership; rewards; and avoiding determinism and reductionism
Final Large Group discussion:
A concern was raised about the danger of over-defining sustainability in higher education: reductionism got us into much of the mess we're in. We must promote a new way of thinking, a new approach. We must start seeing the university as an organic and systemic whole. This is one of our major challenges.
Critical questions from the Sunday meeting:
1.
What do we mean by sustainability?
2.
What critical dimensions, issues, and topics should an institutional case
study in sustainability include? Who is
the reader? Who is the writer? Etc.
3.
What are the best examples of exceptional case studies to date?
4.
What is the purpose of the case study?
Is it for external evaluation? Or is it meant to promote an internal
learning process? Is the
consultant/evaluator a “critical friend” or the “eco-police”?
5.
How can assessments and case studies best be used to promote higher
education for sustainability?
Minutes for advisory group consultation part 1 edited by: Wynn Calder and Peter Blaze Corcoran
Tuesday, October 9
Attendees: Dina
Berzina, Heloise Buckland, Kathy Cacciola, Wynn Calder, Peter Blaze Corcoran,
Sin Declerc, Sokorro Delgado, Peter Maarleveld, Rogier van Mansvelt, Niko
Roorda, Kim Walker, Judy Wilkinson
Chair: Peter Blaze Corcoran
General focus of the meeting: Presentation on the Dutch AISHE method for evaluating sustainability in higher education; and presentation and further discussion of a case study model.
1st Presentation:
Two parts: a) General
description of organizations and networks promoting sustainability in higher
education; b) AISHE – Auditing Instrument for Sustainability in Higher
Education: first results of assessments and policy development in the
Netherlands and Sweden; Niko Roorda, Brabant University for Professional Education,
The Netherlands
1. General description:
a.
Review of
International Networks:
Copernicus
Campus (Copernicus Declaration)
Essence
(environmental education- law to technology)
EEE
Network (environmental engineering education)
b.
Review of global
organizations:
ULSF- University Leaders
for a Sustainable Future (Talloires Declaration)
IAU- International Association of Universities (Kyoto Declaration)
UNESCO- United Nations….
(Agenda 21, Chapter #36)
c.
Global Partnership:
GHESP- Global Higher Education for Sustainability Partnership (ULSF, IAU,
UNESCO, COPERNICUS Campus)
d.
European Partnership:
Copernicus, Essence and EEE Network
e.
U.S.
organization: National Wildlife
Federation’s Campus Ecology Program; Second Nature
f.
Overview of assessment
tools: see Mike Shriberg’s paper (from March 2001 ULSF Consultation; to be
published in joint issue of Higher Education Policy and the International
Journal of Sustainability in Higher Education).
g.
Decision Tree: plans
to develop overview of available assessment tools to facilitate decision-making
process.
h.
CDHO working group,
Netherlands: working on refining AISHE model; plans to offer consultation.
2. AISHE:
a.
Internal assessment
tool, not prescribed but giving opportunity to explore indicators
b.
3 segments of
evaluation: plan, do, check
c.
24 indicators with
rating scale of 5 possibilities: 1 = activity oriented; 2 = process oriented; 3
= system oriented; 4 = chain oriented; 5 = society oriented (e.g. indicator -
staff development plan: 1 = individual initiatives, ad hoc actions; 2 =
short-term plan, budget available; 3 = long-term plan, systematic evaluation
and feedback; 4 = exchange of expertise; 5 = trend-setting in staff development
in sustainability and influence and feedback from society. Note: each stage encompasses the previous
level, 1 being the lowest, 5 being the highest)
d.
Other indicators:
vision and policy, environmental management, professional profile, integration
of curriculum, training, student satisfaction, certification, etc.
e.
Primarily education
focused, with link to operations in item 3.1, EMS (integrating hands-on
learning of operations)
f.
Process: 10-15
students, staff, faculty and administrators individually and subjectively fill
out the survey, using rating scale (quantitative) and writing comments
(narrative, qualitative- verbal description of present situation and
priorities); compile results; consensus meeting to discuss results; input
numbers in computer, which provides graphic representation of current state and
desired state (4-6 hour process).
g.
Working group
discussion: different frameworks fit for different universities; each
institution needs to ID their aims before choosing an assessment tool (e.g.
compare to others or advance campus in certain area).
2nd Presentation:
The University Leaders for a Sustainable Future (ULSF) Advisory Group
on Evaluation and Research; Wynn Calder, Associate Director ULSF, USA; Peter
Blaze Corcoran, Florida Gulf Coast University, USA; and Kim Walker, Consultant,
USA.
1. Review of meeting on Sunday, 10/7
(Calder): Review of previous group discussions: existing and emerging
assessment tools; refinement and development of case study methodology (see
introduction).
2. Case Study Model
(Walker):
a.
Case studies have been
used as a key research strategy in the work being done in sustainability in
higher education. However, they have not necessarily fulfilled their aims
because there are no consistent methods of framing them, thus making it
difficult to compare universities or to learn from the stories told in the case
studies. Moreover, most case studies are used as an assessment tool, written
after the initiative has been completed rather than during the implementation
process. (See Appendix I for a brief discussion of the case study methodology
in sustainability in higher education.)
b.
Purpose of the case
study: the case study may be used as an assessment tool by outside funders or
other outside agencies, or internally, as a means of judging the success of an
initiative. It is also a useful means of sharing stories that others may learn
from. As an assessment tool, it is useful in conjunction with other more
quantitative instruments in that it allows for triangulation of different
perspectives.
c.
Proposal: It is
proposed that a theoretical framework based upon "action research" is
developed. Discussion during the meeting focused on a handout of a draft
framework of a case study model (see Appendix II).
d.
The working group
discussion on the draft framework made the following suggestions: add questions
pertaining to the desired learning outcomes and what is at stake (#1); the
writer must clarify how they are defining sustainability in the context of the
case study (#3); merge items #4 and #5; link to other assessment tools, need
quantitative and qualitative data (#8); if the plans were revised, clearly
outline when, why and by whom (#9).
Minutes for advisory group consultation part
2 recorded by: Kathy Cacciola, National
Wildlife Federation’s Campus Ecology Program Coordinator, USA.; edited by Wynn
Calder.
APPENDIX I:
Case Study Methodology in Sustainability in Higher Education
(Draft)
Kim Walker and Peter Blaze Corcoran
Case study methodology is
a common research tool used in studies of sustainability in higher education.
The field of sustainability in higher education is complex: there are no two
institutions alike, and within institutions, no two schools alike. Moreover,
the concept of sustainability itself is complex. It has been discussed as an ill-defined concept, as a paradigm, as an
integrating, canvassing or heuristic device or, simply stated, as a subject of
normative and ethical discussions.
In universities sustainability
presents an opportunity to make education more problem-based, more
interdisciplinary and more applied. Some value its broad-based international
political clout in bringing environmental issues back to the forefront. Others
see sustainability as a way to improve the image of the university and as a
means to increase enrolments. Again others see in sustainability an opportunity
to reflect on the role the university has to play in society and in the lives
of employees and students, for instance, the university as a healthy and health
promoting community of learners. At the same time voices can be heard that are
rather critical of the sustainability trend. Some even suggest it is a
dangerous hype that masks power struggles and ideological differences (see for
examples: Hesselink et al. (in press); Wals & Jickling (in press)).
Recent literature
on the integration of sustainability in higher education (Corcoran, 1999; van
den Bor et al., 2000) suggests that one can look through different lenses in
trying to come to grips with the concept of sustainability. Most obviously, one
can look at sustainability from an economic perspective, a social perspective
or an ecological perspective. Furthermore, sustainability has different
dimensions that need to be considered: dimensions of space, time, culture and
normative or ethical consideration. In other words, the concept of
sustainability differs from country to country, culture to culture, develops
over time and is based on varying sets of norms and values.
According to Yin (1988, p. 82) case studies allow a researcher to “reveal the multiplicity of factors [which] have interacted to produce the unique character of the entity that is the subject of study.” It is a method of learning about a complex instance through description and contextual analysis. The result is a description and theorizing about why the instance occurred as it did, and what may be important to explore in similar situations. It is:
an empirical study that investigates a contemporary phenomenon within its real-life context; when the boundaries between the phenomenon and context are not clearly evident; and in which multiple sources of evidence are used. (Yin, 1989, p. 23)
Case study methodology, therefore, is the ideal research tool to investigate sustainability in higher education. The case study approach allows the researcher to ‘go deep’, to learn what works and what does not.
Case study research has many differences depending on the purpose of the study, the size of the study, the people involved, the theories developed and the theories tested. Bassey (1999), for example, defines a range of purposes for educational case studies that include theory-seeking and theory-testing case study, story-telling and picture drawing case study and evaluative case study. Case studies may involve description, explanation, evaluation and prediction (Thomas, 1998). Many case studies involve people working within their regular environment.
Case study methods in
sustainability in higher education vary according to the researcher’s purpose
in conducting the case and may involve description, explanation, evaluation and
prediction. Frequently, the researcher is an outside evaluator or critical
friend who sets out to critique the practices of an institution. The aim here
is multi-purpose and often has internal and external purposes. Internally, the
evaluator provides important feedback to the practitioners involved in an
innovation and often works with these people to move forward. The evaluator may
provide feedback to the institution as a whole in the form of a report on the
success or otherwise of the implementation of sustainability in the
institution.
Externally, the evaluator
may compare institutions in an effort to identify practices that work and those
that do not. This work is particularly valuable for those who are attempting
reform in their own institutions. The work also provides important data for
funding groups and potential funding groups.
The case studies may be
conducted by the practitioners involved in the innovation. Here the aim is to
engage in a self study of their own practices. While this form of case study
has the disadvantage of not providing critical external feedback it is a
valuable tool in improving practices.
Whether an outside evaluator is conducting the case or insider practitioners are conducting a self study, they are striving for a holistic understanding of cultural systems of action (Feagin, Orum, & Sjoberg 1990). Cultural systems of action refer to sets of interrelated activities engaged in by the actors in a social situation. It is a system of action rather than an individual or group of individuals. This means that the researcher considers not just the voice of individual actors, but also of the relevant groups of actors and the interaction between them.
Bassey, M. (1999). Case Study Research in Educational settings. Buckingham: Open University Press.
Corcoran, P.B. (1999). Assessment of the professional needs of faculty in college and university environmental programs. Paper presented at the American Education Research Association conference, Montreal, 21 April.
Feagin, J., Orum, A., & Sjoberg, G. (Eds.), (1991). A case for case study. Chapel Hill, NC: University of North Carolina Press.
Hesselink, F., van Kempen, P.P. & A.E.J. Wals (in press). ESDebate: International On-line Debate on Education for Sustainable Development. Gland, Switzerland: IUCN.
Robinson, V.M.J. (1993). Problem-based Methodology: Research for the Improvement of Practice. Oxford: Permagon.
Thomas, R. (1998). Conducting Educational Research: A Comparative View. Westport, CO: Bergin and Harvey.
Van den Bor, Holen, P. & Wals, A.E.J. (2000). Sustainability in Higher (Agricultural) Education. In: Van den Bor, Holen, P. Wals, A.E.J. & Filho, W. (Eds.) Integrating Concepts of Sustainability into Education for Agriculture and Rural Development. Frankfurt/M: Peter Lang Publishers.
Wals, A.E.J. & B. Jickling (in press). Process-based Environmental Education: Setting Standards without Standardizing. In: Jensen, B.B., Schnack, K. & V. Simovska (Eds.) Critical Environmental and Health Education. Copenhagen: Royal School of Educational Studies.
Yin, R. (1993). Applications of case study research. Beverly Hills, CA: Sage Publishing.
Yin, Robert K. (1989). Case Study Research: Design and Methods. Beverly Hills, CA: Sage.
APPENDIX II:
Case Study Model: Sustainability in Higher Education
(Draft)
· Who initiated the study?
· What is the purpose of the study?
· What are the anticipated outcomes of the study?
· What is the scope of the study?
· What are the resources available to conduct the study?
· Who is the audience for the final report?
· What are the anticipated format and contents of the final report?
Mission, institutional structure and planning
Curriculum and academic disciplines
Faculty and student research
Institutional operations
Student opportunities
Faculty and staff development and rewards
Service and outreach to local community and beyond
· Any unexpected outcomes?
E.g., Funding
Staff development
Management support
Community outreach
Career counseling for students
Sustainability committee/s, working parties
***********************************************