The problem addressed in this tutorial is the lack of a reliable methodology to determine what has been learned in problem-centered learning environments for complex domains. Large investments are being made in the area of problem-centered learning, especially when technology is involved in the delivery of instruction. Have these investments in educational research and technology been worthwhile? Many have probably contributed to improved learning (Kulik, 1994; Sivin-Kachala & Bialo, 2000), although the major finding in studies investigating the relative effects of new approaches and new technologies on learning is typically no significant difference (Russell, 1999).

Common measures of learning reported in the literature include scores on standardized tests, subjective indications of interest and attitude, and identification/recall of concepts and simple procedures (Becker, 2000; Gabel, 1994; Koszalka, 2002). While such measures provide indirect evidence of improvement in higher-order learning, they may not correlate with improved understanding in complex domains (Grotzer & Perkins, 2000). It is not generally known how best to support learning in and about complex domains (Doran et al., 1994; Dörner, 1996; Sterman, 1994). It is generally believed that an experiential, problem-centered environment with rich opportunities for collaboration with peers will be effective. Such an approach has been reflected in the most promising systems developed in the 1990’s (Brandt, 2000; Feldman et al., 2000). However, these collaborations break down when students are asked to derive hypotheses from data or engage in serious reflective discourse (i.e., exemplify higher-order thinking). Moreover, well-educated and highly motivated adult learners have difficulty in making strategic decisions about complex systems (Dörner, 1996).

he DEEP methodology demonstrated in this tutorial is based on a view of learning as becoming more expert-like (Ericsson & Smith, 1991) and more skilled in higher-order causal reasoning and problem-solving (Grotzer & Perkins, 2000). The learning-as-becoming-like-an-expert perspective treats learning as a continuing process of growth rather than a single end-point measurable by a simple test. The DEEP methodology involves (1) identifying characteristic problems for problem-centered instructional modules; (2) eliciting both expert and novice patterns of problem solving; (3) representing these patterns in a standard form; (4) measuring similarities and differences among experts and novices; and, (5) assessing changes in problem-solving performance patterns over time and with experience. This process provides a measure of higher-order problem-solving ability as well as a way to track and analyze individual learner change and progress. Three indicators of change and potential progress towards expertise are: (1) surface similarity; (2) structural similarity; and, (3) semantic similarity. These measures will be collected to see how individuals change and whether and how responses begin to resemble expert responses.

The tutorial is intended for all those who teach, develop instruction, evaluate outcomes and conduct research on learning in complex domains, which are typified by problem situations involving multiple acceptable solutions (e.g., environmental decision making, engineering design, medical diagnosis, etc.). Instruction in these domains often involves modelling and simulation activities but learning outcomes are often assessed using simpler methods such as knowledge-based tests. The inadequacy of these methods and the need for more authentic performance-based methods will be presented. Participants will learn how to use modelling and simulation activities to support assessment and determine relative level of expertise. The same tools and methods can and have been used to support instruction in these domains.

00.00 - 00.15 Introduction, goals and objectives
00.15 – 01.00 The nature of learning and the nature of complex domains
01.00 - 01.15 Short activity to describe a learning task in a complex domain
01.15 - 01.30 Discussion of how to support learning such tasks and how to assess outcomes
01.30 - 02.00 (Break and informal discussion)
02.00 - 02.45 A system dynamics perspective of complexity and assessment
02.45 - 03.00 Short activity to develop an assessment for tasks described earlier
03.00 - 03.30 Demonstration of the DEEP methodology and discussion of findings

This tutorial involves an overview of the research literature pertaining to learning in and about complex domains, with an emphasis on problem-centered learning, model-facilitated instructional approaches and a system dynamics analysis of complex domains. Demonstrations of several modelling and simulation tools will be presented. An explanation of assessment methods appropriate for these domains will then be discussed and demonstrated. Consequently, the primary method of presentation will be expository with interactions actively solicited from participants. Several periods for participant interaction in applying principles and methods discussed will be used to stimulate discussion and explore variations to assessment methods presented.

It is this author's experience that questions and debate contributions tend to arise after the presentation of particular ideas rather than during planned discussion periods. Accordingly, the only time specifically allocated for discussion will be at the end of the tutorial; but the amount of material to be presented will be adjusted so that an average of 5 minutes of information discussion and debate can take place within each of the longer sections of the tutorial, where the need arises.