[LON-CAPA-cvs] cvs: modules /gerd/roleclicker description.tex
www
lon-capa-cvs@mail.lon-capa.org
Wed, 18 May 2005 19:41:08 -0000
This is a MIME encoded message
--www1116445268
Content-Type: text/plain
www Wed May 18 15:41:08 2005 EDT
Modified files:
/modules/gerd/roleclicker description.tex
Log:
Mnay changes
--www1116445268
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20050518154108.txt"
Index: modules/gerd/roleclicker/description.tex
diff -u modules/gerd/roleclicker/description.tex:1.61 modules/gerd/roleclicker/description.tex:1.62
--- modules/gerd/roleclicker/description.tex:1.61 Wed May 18 13:45:29 2005
+++ modules/gerd/roleclicker/description.tex Wed May 18 15:41:05 2005
@@ -118,7 +118,7 @@
\section{Relevant Results from Prior NSF Support}\label{results}
\subsection{MSU Group}\label{priormsu}
-Gerd Kortemeyer is PI on the current NSF-ITR grant Investigation of a Model for Online Resource Creation and Sharing in Educational Settings (\#0085921, \$2,055,000, September 15, 2000 through July 31, 2005). The project
+Gerd Kortemeyer is PI on the current NSF-ITR grant {\it Investigation of a Model for Online Resource Creation and Sharing in Educational Settings} (\#0085921, \$2,055,000, September 15, 2000 through July 31, 2005). The project
developed the cross-institutional learning content management system Learning{\it Online} Network with Computer Assisted Personalized Approach (LON-CAPA) and researched methods to assess the educational impact of
content resources and representations within its shared content pool.
@@ -129,20 +129,9 @@
LON-CAPA is designed around the concept of easy sharing and re-use of learning resources, as well as advanced assessment capabilities in science courses.
The system started in 1992 as a system to give personalized homework to students in introductory physics courses. ``Personalized" means that each student sees a different version of the same
computer-generated problem: different numbers, choices, graphs, images, simulation parameters, etc, Fig.~\ref{rando}.
-\begin{figure}[t]
-\includegraphics[width=8cm]{emfRand1}
-\includegraphics[width=8cm]{emfRand3}
-\includegraphics[width=8cm]{collRand2}
-\includegraphics[width=8cm]{collRand3}
-\caption{\small Three randomized variations of the problems in Fig.~\ref{repre} and \ref{reprecoll}.
-The graphs in the emf problem are dynamically generated on-the-fly, the cars and the brickwall in the collision problem are randomly selected images.
-Each randomization leads to different answersfor different students. It should be noted that in this particular example, the collision problem
-became more difficult if the learner realized early on that in all combinations of Fig.~\ref{reprecoll}, the combined object after the collision will be at rest
--- the corresponding constraints could have been implemented in the randomizing problem.\label{rando}}
-\end{figure}
Over the years, the system added a learning content management system and standard course management features, such as communications, gradebook, etc., which are comparable to commercial course management systems, such as
-BlackBoard, WebCT, or ANGEL. SeeRefs.~\cite{features,edutools} for an overview of features, and comparisons to other systems.
+BlackBoard, WebCT, or ANGEL. See Refs.~\cite{features,edutools} for an overview of features, and comparisons to other systems.
In addition to standard features, the LON-CAPA delivery and course management layer is designed around STEM education, for example:
support for mathematical typesetting throughout (\LaTeX\ inside of XML) -- formulas are rendered on-the-fly,
and can be algorithmically modified through the use of variables inside formulas; integrated GNUplot support, such that graphs can be rendered on-the-fly,
@@ -159,41 +148,38 @@
\caption{\small \label{fig:gradecorrel}Prominance of discussion contribution characteristics by student grade (left panel) and question difficulty (right panel).}
\end{figure}
-{\bf Student Course Grade:} Significant differences as a function of course grade appeared when considering the classes of discussions (subsection~\ref{subsec:disccat}).
-Figure~\ref{fig:gradecorrel} shows the outcome of this study by discussion superclass. As an example, the figure is to be interpreted this way: within the indicated errors,
+\noindent{\it Student Course Grade:} The left panel of
+Figure~\ref{fig:gradecorrel} shows the character of online student discussion contributions over the semester as a function of final grade in the course.
+As an example, the figure is to be interpreted this way: within the indicated errors,
55 percent of a 3.0 student's discussion contributions were solution-oriented.
-The lines represent second-order polynomial fits to the data. The results are not surprising, but verify the validity of the classification approach, which will also be used for this project.
+The results are not surprising, but verify the validity of the classification approach, which will also be used for this project.
At the same time, the results confirm that conceptual and physics-related discussions are positively correlated with success in the course, while solution-oriented discussion contributions are strongly negatively correlated.
While cause and effect may be arguable, in this project, particular attention needs to be paid to question properties that elicit either the desirable or undesirable discussion patterns.
-
-{\bf Influence of Question Types:} Different question types were found to result in different associated discussion patterns. Discussions on a procedural level are more prominent for numerical problems than for any other
+\newline
+{\it Influence of Question Types:} Discussions on a procedural level were found to be more prominent for numerical problems than for any other
problem type. Solution-oriented discussions are more prominent for multiple-choice style questions, frequently in an effort to short-circuit the conceptual reasoning: it was found that students in this
simple question type use the discussion space to reverse-engineer the randomization process by copying-and-pasting the their correct solutions.
The prominance of conceptual discussions is significantly lower in single-response multiple-choice (the type currently used in Peer Instruction)
and numerical problems than in the other problem types. In the earlier study by Kashy~\cite{kashyd01}, it was also found that mastery of these same question types does not predict overall performance on the final exam
as well as other question types. Multiple-choice problems that do not involve numbers are frequently simply called ``conceptual'' questions, but in~\cite{discpaper}, it was found that they do not {\it necessarily} lead to
conceptual discussions. Ranking questions showed very favorable discussion patterns, but their sample size in~\cite{discpaper} has been too small to make definitive statements.
-Differences between ``conventional'' and representation-translation problems were small, while Kashy~\cite{kashyd01} found significant differences,
-and McDermott~\cite{mcdermott} and Beichner~\cite{beichner} pointed out the significance of the translation step. It should be noted that the earlier study~\cite{kashyd01} dealt with a relatively small set of
-representation-translation problems, some of which involved non-static time-evolving simulations as data-source, while in~\cite{discpaper}, none of the simulation-based problemswere assigned. Within this project, we aim to deploy
-Physlets~\cite{physlets} in the classroom, and expect statistically more significant data regarding their impact.
-
-{\bf Influence of Question Difficulty:} The right panel of Fig.~\ref{fig:gradecorrel} shows the prominance of different discussion contributions as a function of questiondifficulty.
+\newline
+{\it Influence of Question Difficulty:} The right panel of Fig.~\ref{fig:gradecorrel} shows the prominance of different discussion contributions as a function of questiondifficulty.
Very easy problems can elicit a high level conceptual discussions, and so can problems of mid-range difficulty. As problems become more difficult, there is no significant gain in conceptual discussions.
-An important feature of LON-CAPA for this study is its scalability and ability to absorb peak-workloads through cluster-wide load balancing. On campus of MSU, LON-CAPA has been used for timed online ``take-home'' exams by 400 students submitting 50 questions each within one hour.
-
-The project developed into a content sharing network of more than 15 institutions of higher education including community colleges and four-year institutions, as well as over 15 middle and high schools. In addition, LON-CAPA houses commercial textbook content from seven major publishing companies, and a commercial service company was established around the product at the end of 2004. The project maintains a gateway server to the National Science Digital Library, and the LON-CAPA shared resource pool is searchable and accessible from {\tt http://nsdl.org/}.
+The project developed into a content sharing network of more than 15 institutions of higher education including community colleges and four-year institutions, as well as over 15 middle and high schools. In addition, LON-CAPA houses commercial textbook content from seven major publishing companies, and a commercial service company was established around the product at the end of 2004.
+The shared content pool currently contains over 60,000 learning content resources, including more than 18,000 personalized homework problems.Disciplines include astronomy, biology, business, chemistry, civil engineering, computer science, family and child ecology, geology, human food and nutrition,human medicine, mathematics, medical technology, physics, and psychology.
+The project maintains a gateway server to the National Science Digital Library, and the LON-CAPA shared resource pool is searchable and accessible from {\tt http://nsdl.org/}.
LON-CAPA is used by approximately 25,000 students every semester.
-Gerd Kortemeyer is Co-PIs on the current NSF-CCLI-ASA grant Diagnostic Question Clusters: Development and Testing in Introductory Geology and Biology (\#0243126, \$491,606, September 15, 2003 through August 31, 2006) to develop diagnostic questions for college students in both biology and geology. In the ASA project, a pool of peer-reviewed, diagnostic question clusters to assess students' understanding will be developed, including tools for analysis, peer review, and online publication of these question clusters.
+Gerd Kortemeyer is Co-PIs on the current NSF-CCLI-ASA grant {\it Diagnostic Question Clusters: Development and Testing in Introductory Geology and Biology} (\#0243126, \$491,606, September 15, 2003 through August 31, 2006) to develop diagnostic questions for college students in both biology and geology. In the ASA project, a pool of peer-reviewed, diagnostic question clusters to assess students' understanding will be developed, including tools for analysis, peer review, and online publication of these question clusters.
\subsection{Harvard Group}
The group of Eric Mazur has been supported by the National Science Foundation to develop, evaluate, and produce
materials for effective undergraduate science pedagogy and assessment since 1993
-(DUE \#925407, Peer Instruction: Stimulating renewed interest in physics and other
-science and engineering courses,
+(DUE \#925407, {\it Peer Instruction: Stimulating renewed interest in physics and other
+science and engineering courses},
\$223,500,
-2/1/93 - 1/31/96).
+2/1/93-1/31/96).
We have applied Peer Instruction in both the calculus-based and the algebra-based introductory physics courses for non-majors at Harvard University.
Instructors have adopted the method across a variety of disciplines and courses, including senior-level courses, at a large number of institutions nationwide. Substantial gains in
@@ -220,11 +206,11 @@
The primary resource needed for teaching with Peer Instruction is a supply of suitable ConcepTests (CTs) -- questions that test students' understanding of the basic concepts covered \cite{mref11}. We have developed and
refined over 1,000 CTs for use in introductory physics courses. These CTs are freely available to instructors through the ILT web site (detailed below), together with over 400 additional CTs that have been contributed by others.
An indicator of the rapid spread of the method is the availability of books with ConcepTests for chemistry, astronomy, and calculus courses.
-The group developed Project Galileo (DUE \#9554870, On-line server of educational resources, \$800,000, 3/1/96-3/1/00; DUE \#9980802 Creating a community of Peer Instruction users: dissemination and
-electronic resources, \$290,000, 4/01/00-3/31/02), as a store of extensive resources for these
+The group developed Project Galileo (DUE \#9554870, {\it On-line server of educational resources}, \$800,000, 3/1/96-3/1/00; DUE \#9980802, {\it Creating a community of Peer Instruction users: dissemination and
+electronic resources}, \$290,000, 4/01/00-3/31/02), as a store of extensive resources for these
interactive learning pedagogies, targeting both large and small classroom teaching techniques, which are available to the entire teaching community.
- Using funds from a NSF Director's Distinguished Teaching Scholar Award (DUE \#123899, Distinguished Teaching Scholar Award: On-line Resources for Teaching With Peer Instruction, \$305,000, 9/15/01-8/31/05),
+ Using funds from a NSF Director's Distinguished Teaching Scholar Award (DUE \#123899, {\it Distinguished Teaching Scholar Award: On-line Resources for Teaching With Peer Instruction}, \$305,000, 9/15/01-8/31/05),
we created the Interactive Learning Toolkit (ILT), a learning management system that allows instructors to implement several proven innovative teaching techniques and to share and review materials they create for these techniques.
The ILT features a set of tools that allow an instructor to structure and create content for their class and then analyze the feedback of the students.
In order to 'free up' precious class time, the ILT offers an pre-class reading assignment tool.
@@ -251,70 +237,122 @@
functionality toward a variety of broadly available commodity devices with standard two-way communication protocols. The resulting system can be used in variety of classroom systems, but due to the usage of standard communication
protocols, also in distance learning situations,
where participant groups can be located at different locations. The project makes use of a distributed shared content pool, which enables scalable dissemination of content from participating instructors and institutions.
-\section{Methodology}\label{method}
-The effectiveness of the extensions to Peer Instruction will be evaluated both with focus on process and on learning outcomes.
+\section{Expansion of Technology Infrastructure}
+\begin{figure} [t]\begin{center}\includegraphics[width=4.5in]{overview.eps}\caption{\small Overview of the implementation process.\label{overview}}\end{center}\end{figure}
-\subsection{Pre-Tests and Assessment Instruments}
-\label{inventories}
-The project involves three academic institutions of very different character. In order to compare and contrast results gathered across these institutions,
-it is important to assess attitutides and prior subject knowledge of the different student populations in order to compare and contrast research findings.
+In order to carry out the research proposed, new and existing technology components need to be combined into an expanded infrastructure, which the PIs will use at the three instutitions involved.
+While still researching their effectiveness and impact, these technology components are loosely linked,
+while in the final stage of this project, successful components will be tightly integrated into a software package that can be broadly disseminated (Fig.~\ref{overview}).
-Instruments have been developed to assess epistemological beliefs, for example the Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument~\cite{EBAPS}.
-Related to epistemological beliefs are learners' expectations and attitudes, and of particular interest to research in physics education is the Maryland Physics Expectations (MPEX) survey~\cite{MPEX}.
-
-To assess prior knowledge of the subject area, we will use existing concept inventory surveys as both pre- and post-tests.
-The qualitative Force Concept Inventory~\cite{fci} and the quantitative companion Mechanical Baseline Test~\cite{hestenesmech} have been used in a large number of studies connected to the teaching of introductory mechanics.
-The Foundation Coalition has developed a number of concept inventories~\cite{foundation}, namely the Thermodynamics Concept Inventory, the Dynamics Concept Inventory, and the Electromagnetics Concept Inventory (with two subcomponents, namely Waves and Fields). In addition, the Conceptual Survey of Electricity and Magnetism (CSEM)~\cite{maloney} is available for the second semester course.
-\subsection{\label{subsec:problemcat}Problem Classification}
-Kashy~\cite{kashyd01} showed that student mastery of different types of homework problems correlates differently with the students' performance on final exams ---
-with multiple-choice non-numerical problems having the lowest correlation, and numerical/mathematical problems that require a translation of representation having the highest.
-Steinberg~\cite{steinberg} also analyzed student performance on multiple-choice diagnostics and open-ended exam problems, and found that while those correlate in general, for certain students
-and certain questions, responses differ greatly.
-For this project, we are choosing a finer-grained classification scheme of question types (adapted from Redish~\cite{redish}):
+In the initial phase, the Interactive Learning Toolkit will be integrated with BQ, forming a new software package called LT3 (Learning Together Through Technology). At the moment, the
+combined software already allows to collect and analyze the PRS responses from Peer Instruction using BQ in class.
+Instructors will be able to run LT3 in three formats: (a) completely off a server, (b) completely off the instructors server or (c) in a modularized from, where the
+ILT functionality is kept on a server and the Interactive Classroom component is running off a computer in the classroom. After initial tests, option (c) has been
+found to be the most optimal solution for a large classroom environment. In this solution, instructors continue to prepare the content for the class by browsing
+the CT data base and integrate CTs into a given lecture. The CTs are then uploaded to the computer running the Interactive Classroom portion of the LT3. After each
+lecture, student results are uploaded to the server part of LT3 and made available for statistical and individual performance analysis.
+
+\subsection{Implementation of the Resource Pool}
+In LON-CAPA, the underlying distributed multimedia content repository spans across all participating institutions.
+Any content material contributed to the pool is immediately available and ready-to-use within the system at all participating sites, thus facilitating dissemination of curricular development efforts.
+A large fraction of these resources are also available through the gateway to the National Science Digital Library (NSDL).
-\noindent{\bf Single-Response Multiple-Choice:} The most basic and most easily computer-evaluated type of question, where only one option is correct,
-see for example the original ConcepTest problems on the left of Figs.~\ref{repre} and \ref{reprecoll}.\newline
-{\bf Multiple-Response Multiple-Choice} This type of problem, a first step beyond single-response problems, requires a student to evaluate each statement and make a decision about it. The problem on the right side of Fig.~\ref{repre} is of this type.
-\newline
-{\bf Numerical and Formula Short Answer:} Numerical answers (potentially in multiple dimensions and including physical units), such as ``\verb!17 kg/m^3!" or mathematical expressions (potentially in multiple dimensions), such as ``\verb!1/2*m*(vx^2+vy^2)!", are expected. \newline
-{\bf Ranking-Tasks:} This type of problem requires a student to rank a number of statements, scenarios, or objects with respect to a certain feature. For example, a student might be asked to rank a number of projectiles in the order that they hit the ground, or a number of locations in order of the strength of their local electric potential.
-Several options may have the same rank (``tie''). The left panel of Fig.~\ref{reprecoll} is of this type.
-\newline
-{\bf Click-on-Image:} Learners need to click on different parts in an image, for example on where to cut a wire in order to brighten up a lightbulb elsewhere in a circuit diagram.
+The ConcepTest library will be ported to the LON-CAPA, which offers scalable cross-institutional content and rights management features. Where appropriate, gateways will be established to have different
+system components access the same original content. The LT3 will be enabled to search the LON-CAPA CT resource pool and to render a static or randomized version into the LT3 environment. This will enable the LT3 as
+well as the LON-CAPA community to access the most recent version of each ConcepTest, as well as to track any changes which have been made by the individual authors. The common content pool
+allows us to carry out the proposed study with a minimum of logistical or technological challenges caused by the involvement of three different institutions.
-\begin{figure} [t]
-\includegraphics[width=8cm]{emfOrig}
-\includegraphics[width=8cm]{emfMulti}
+\subsection{Computer-Guided Group Formation} \label{groupform}
+The goal of computer-guided group formation is to generate student groups with differing initial responses.
+Group formation is limited by seating arrangements in a the lecture hall, unless time can be afforded for students to get up and walk around the lecture hall.
+As a first step, seating arrangements in the lecture hall need to be coded into a computer-readable format -- both the BQ and the LON-CAPA group have experience in this area.
+Then an algorithm needs to be developed to find the optimum configuration of groups of $N$ nearest neighboring students that maximizes initial dissent within the groups.
+Fig.~\ref{formation} is a mock-up of a possible configuration of two nearest neighbors within a lecture hall.
+We will implement a hybrid scheme, where internet based two-way communication, telling the individual student where to turn, will be combined with
+a projection of the map to the front of the class . This implementation is very flexible, since it can be used in almost any teaching environment independent of the technological resourcs available.
+\begin{figure}[t]\begin{center}\includegraphics[width=2.2in]{seatfig.eps}\caption{\small Computer-guided group formation.\label{formation}}\end{center}\end{figure}
+The system records group configurations and makes analyses of pre- and post-discussion responses within the groups possible.
-\caption{\small Example of two problems addressing the same concepts in two different representations. The problem on the left is an original ConcepTest problem of type single-response multiple-choice, the problem on the right is multiple-response multiple-choice. Both problems require representation translation since data is provided in graphical form.\label{repre}}
-\end{figure}
+\subsection{Implementation of the Resource Pool}
+In LON-CAPA, the underlying distributed multimedia content repository spans across all of the currently over 30 participating institutions, and currently contains over 60,000 learning content resources,
+including more than 18,000 personalized homework problems.
+ Disciplines include astronomy, biology, business, chemistry, civil engineering, computer science, family and child ecology,
+geology, human food and nutrition, human medicine, mathematics, medical technology, physics, and psychology.
+Any content material contributed to the pool is immediately available and ready-to-use within the system at all participating sites, thus facilitating dissemination of curricular development efforts.
+A large fraction of these resources are also available through the gateway to the National Science Digital Library (NSDL).
-In addition, we consider the following features, which may or may not apply to any question type (adapted from Redish~\cite{redish}):
+The ConcepTest library will be ported to the LON-CAPA, whichoffers scalable cross-institutional content and rights management features.
+Where appropriate, gateways will be established to have different system components access the sameoriginal content.
-\noindent
-{\bf Representation-Translation:} This type of surprisingly challengingi~\cite{mcdermott,beichner} problem requires a student to translate between different representations of the same situation, for example from a graphical to a numerical or textual representation. The answer might be required in different formats, see for example
-Fig.~\ref{repre}.
-\newline
-{\bf Context-based Reasoning:} The distinguishing characteristic of these problems is that they are set in the context of real-world scenarios and not in the context of the artificial ``zero-friction" laboratory scenarios of typical textbook problems.
+\subsection{\label{subsec:problemcat}Different Question Types}
+\begin{figure}\includegraphics[width=3.5in]{dell.eps}\includegraphics[width=2.7in]{sharp2.eps}\caption{\small Rendering of a problem on PDA devices\label{pdaview}}
+\end{figure}
-The same classification scheme was successfully used for an analysis of online asynchronous discussions associated with online homework problems~\cite{discpaper}.
-\subsection{Process-Oriented Evaluation}
-The process-oriented evaluation will focus on the actual discussion process. Since currently no baseline data exists for this study, we will assess the quality of student discussion both
-before and after the introduction of extensions to the current Peer Instruction technique.
-The PIs of this project have experience analysing student discussions from a prior project which examined asynchronous online student discussions around different types of online homework problems.
-After categorization of both the problem types and the discussion contributions, significant differences in the student discussion behavior around different problem types were extracted (subsection~\ref{priormsu}).
+LON-CAPA currently has the ability to present a wide variety of question types, which we will classify as follows
+ (adapted from Redish~\cite{redish}):
+\noindent{\bf Single-Response Multiple-Choice:} The most basic and most easily computer-evaluated type of question, where only one option is correct,
+see for example the original ConcepTest problems on the left of Figs.~\ref{repre} and \ref{reprecoll}.\newline
+{\bf Multiple-Response Multiple-Choice} This type of problem, a first step beyond single-response problems, requires a student to evaluate each statement and make a decision about it. The problem on the right side of Fig.~\ref{repre} is of this type.\newline
+{\bf Numerical and Formula Short Answer:} Numerical answers (potentially in multiple dimensions and including physical units), such as ``\verb!17 kg/m^3!" or mathematical expressions (potentially in multiple dimensions),
+such as ``\verb!1/2*m*(vx^2+vy^2)!", are expected.
+\newline{\bf Ranking-Tasks:} This type of problem requires a student to rank a number of statements, scenarios, or objects with respect to a certain feature. For example, a student might be asked to rank a number of projectiles in the order that they hit the ground, or a number of locations in order of the strength of their local electric potential.
+Several options may have the same rank (``tie''). The left panel of Fig.~\ref{reprecoll} is of this type.\newline
+{\bf Click-on-Image:} Learners need to click on different parts in an image, for example on where to cut a wire in order to brighten up a lightbulb elsewhere in a circuit diagram.
+\begin{figure} [t]\includegraphics[width=8cm]{emfOrig}\includegraphics[width=8cm]{emfMulti}
+\caption{\small Example of two problems addressing the same concepts in two different representations. The problem on the left is an original ConcepTest problem of type
+single-response multiple-choice, the problem on the right is multiple-response multiple-choice. Both problems require representation translation since data is provided in graphical form.\label{repre}}\end{figure}
\begin{figure} [t]
\includegraphics[width=8cm]{collOrig}
\includegraphics[width=8cm]{collRankNR}
-\caption{\small Example of two problems addressing the same concepts in two different representations. The problem on the left is an original ConcepTest problem of type single-response multiple-choice, the problem o\
-n the right is rank-response.\label{reprecoll}}
+\caption{\small Example of two problems addressing the same concepts in two different representations. The problem on the left is an original ConcepTest problem of type single-response multiple-choice,
+the problem
+on the right is rank-response.\label{reprecoll}}
\end{figure}
+In addition, we consider the following features, which may or may not apply to any question type (adapted from Redish~\cite{redish}):
+\noindent{\bf Representation-Translation:} This type of surprisingly challenging~\cite{mcdermott,beichner} problem requires a student to translate between different representations of the same
+situation, for example from a graphical to a numerical or textual representation. The answer might be required in different formats, see for example Fig.~\ref{repre}.
+\newline{\bf Context-based Reasoning:} The distinguishing characteristic of these problems is that they are set in the context of real-world scenarios and not in the context of the
+artificial ``zero-friction" laboratory scenarios of typical textbook problems.
+
+\subsection{Randomized Questions}\label{randomques}
+\begin{figure}[t]
+\includegraphics[width=8cm]{emfRand1}\includegraphics[width=8cm]{emfRand3}
+\includegraphics[width=8cm]{collRand2}\includegraphics[width=8cm]{collRand3}
+\caption{\small Three randomized variations of the problems in Fig.~\ref{repre} and \ref{reprecoll}. The graphs in the emf problem are dynamically generated on-the-fly, the cars and the brickwall in the
+collision problem are randomly selected images. Each randomization leads to different answersfor different students. It should be noted that in this particular example, the collision problem became more
+difficult if the learner realized early on that in all combinations of Fig.~\ref{reprecoll},
+the combined object after the collision will be at rest-- the corresponding constraints could have been implemented in the randomizing problem.\label{rando}}
+\end{figure}
+The LON-CAPA system is already capable of randomizing questions with a wide range of options and also provides statistical
+tools for the de-randomized analysis of responses.
+For classroom use, specialized tools similar to the existing ILT functionality need to be developed to provide theinstructor with a quick and
+comprehensive overview of response patterns of the more complex randomizing question types.
+
+\section{Research Methodology}\label{method}
+The variables are
+\begin{itemize}\item Computer-Guided Group Formation\item Different question types\item Randomized questions\end{itemize}
+These will be evaluated both with focus on process and on learning outcomes.
+
+\subsection{Pre-Tests and Assessment Instruments}
+\label{inventories}
+The project involves three academic institutions of very different character. In order to compare and contrast results gathered across these institutions,
+it is important to assess attitutides and prior subject knowledge of the different student populations in order to compare and contrast research findings.
+
+Instruments have been developed to assess epistemological beliefs, for example the Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument~\cite{EBAPS}.
+Related to epistemological beliefs are learners' expectations and attitudes, and of particular interest to research in physics education is the Maryland Physics Expectations (MPEX) survey~\cite{MPEX}.
+
+To assess prior knowledge of the subject area, we will use existing concept inventory surveys as both pre- and post-tests.
+The qualitative Force Concept Inventory~\cite{fci} and the quantitative companion Mechanical Baseline Test~\cite{hestenesmech} have been used in a large number of studies connected to the teaching of introductory mechanics.
+The Foundation Coalition has developed a number of concept inventories~\cite{foundation}, namely the Thermodynamics Concept Inventory, the Dynamics Concept Inventory, and the Electromagnetics Concept Inventory (with two subcomponents, namely Waves and Fields). In addition, the Conceptual Survey of Electricity and Magnetism (CSEM)~\cite{maloney} is available for the second semester course.
+\subsection{Process-Oriented Evaluation}
+The process-oriented evaluation will focus on the actual discussion process. Since currently no baseline data exists for this study, we will assess the quality of student discussion both
+before and after the introduction of extensions to the current Peer Instruction technique.
Student discussion entries are classified into four types and with ten possible features. The four types~\cite{discpaper} are
\noindent{\bf Emotional:} discussion contributions were classified as ``emotional" if they mostly communicated opinions,
@@ -335,11 +373,9 @@
In addition, the following features are considered, which were used in
an earlier study of discussions around group exercises in an introductory Computer Science course at Michigan State University (derived from Johnson et al.~\cite{johnson}):
Contributes Idea; Encourages Participation; Summarizes/Integrates; Check for Understanding; Relates New to Old Learning; Gives Direction to Work.
-
-Table~\ref{table:examples} shows an example of a discussion classification according to this scheme.
\begin{table}
\caption{\small Example of a discussion classification around the collision problem Fig.~\ref{reprecoll}.\label{table:examples}}
-\small
+\footnotesize
\begin{tabular}{l|p{8cm}|p{6cm}}
Speaker&Contribution&Classification\\\hline
A&It's "inelastic," so they'll just sit there after the crash.&Conceptual; Physics.\\\hline
@@ -362,12 +398,12 @@
\end{tabular}
\end{table}
-Student helpers will be trained and assigned to student groups during lecture to document the discussions using the coding scheme described in subsection~\ref{subsec:disccat}. They will be provided with worksheets to quickly tabulate contributions. Results will be stored in conjunction with the statistical data gathered from each question. Analysis will be carried out as described in \ref{prevdiscrev}.
+Student helpers will be trained and assigned to student groups during lecture to document the discussions using this coding scheme, see Table~\ref{table:example}.
+As in an earlier study carried out at MSU, helpers will be provided with worksheets to quickly tabulate contributions using tickmarks. While we expect that early in the semester, their presence will influence the discussion process, in a later phase, students get used to their presence. Results will be stored in conjunction with the statistical data gathered from each question, and the analysis will be carried out as described in \ref{priormsu}.
We will interview focus groups of students regarding their experiences and perceived relative helpfulness of the different problem types, and ask them to also reflect on how they perceived these question types were influencing their problem-solving strategies. Pascarella~\cite{pascarella02} developed some frameworks for these interviews, which can be built upon.
\subsection{Outcome-Oriented Evaluation}
-\subsubsection{Pre-/Post-Discussion Answer Distribution}
\begin{figure}[t]
\begin{center}
\includegraphics[width=2in]{before}
@@ -385,84 +421,9 @@
The same concept inventories using the establishment of initial conditions (subsection~\ref{inventories}) will be used in a post-test scenario.
-At Harvard, the FCI will be used in the second semester course, since many concepts taught in the Eletricity and Magnetism Courses are the same as in Mechanics (forces, fields, potential), an improved
-conceptual understanding through a more effective form of peer instruction would lead to an improved performance in both tests.
-Figure~\ref{prepostfci} shows pre- and post scores of three courses at Harvard using a traditional style of lecture, a hybrid form which allows some
-form of interaction during class, and a course using Peer Instruction. Clearly, the performance enhancement is highest for the fully
-interactive course. A more complete picture would also take into account the different forms of Peer Instruction proposed.
-
A capability of our systems is that we can use the same question without modifications in online and bubble-sheet exam mode.
Since in addition, questions are randomizing, we are able to include some of the same questions used in class on exams and
quizzes. A similar study was previously conducted by Kashy~\cite{kashyd01} for homework questions.
-
-\section{Implementation of the Resource Pool}
-
-In LON-CAPA, the underlying distributed multimedia content repository spans across all of the currently over 30 participating institutions, and currently contains over 60,000 learning content resources, including more than 18,000 personalized homework problems. Disciplines include astronomy, biology, business, chemistry, civil engineering, computer science, family and child ecology, geology, human food and nutrition, human medicine, mathematics, medical technology, physics, and psychology. Any content material contributed to the pool is immediately available and ready-to-use within the system at all participating sites, thus facilitating dissemination of curricular development efforts. A large fraction of these resources are also available through the gateway to the National Science Digital Library (NSDL).
-
-The ConcepTest library will be ported to the LON-CAPA system and stored in the shared resource pool of the LearningOnline Network with CAPA (subsection~\ref{loncapa}), which
-offers scalable cross-institutional content and rights management features. Where appropriate, gateways will be established to have different system components access the same
-original content.
-
-\section{Research Phase}
-\begin{figure} [t]
-\begin{center}
-\includegraphics[width=4.5in]{overview.eps}
-\caption{\small Overview of the implementation process.\label{overview}
-}
-\end{center}
-\end{figure}
-
-\subsection{Expansion of Infrastructure}
-In order to carry out the research proposed, new and existing technology components need to be combined into an expanded infrastructure, which the PIs will use at the three instutitions involved. While still researching their effectiveness and impact, these technology components are loosely linked, while in the final stage of this project, successful components will be tightly integrated into a software package that can be broadly disseminated (Fig.~\ref{overview}).
-
-In the initial phase, the Interactive Learning Toolkit will be integrated with BQ, forming a new software package called LT3 (Learning Together Through Technology). At the moment, the
-combined software already allows to collect and analyze the PRS responses from Peer Instruction using BQ in class.
-Instructors will be able to run LT3 in three formats: (a) completely off a server, (b) completely off the instructors server or (c) in a modularized from, where the
-the ILT functionality is kept on a server and the Interactive Classroom component is running off a computer in the classroom. After initial tests, option (c) has been
-found to be the most optimal solution for a large classroom environment. In this solution, instructors continue to prepare the content for the class by browsing
-the CT data base and integrate CTs into a given lecture. The CTs are then uploaded to the computer running the Interactive Classroom portion of the LT3. After each
-lecture, student results are uploaded to the server part of LT3 and made available for statistical and individual performance analysis.
-
-In parallel, the CT library will be transfered to the LON-CAPA network as described above. The user interface for CT browsing and lecture integration in the LT3 will remain
-unchanged. The LT3 will be enabled to search the LON-CAPA CT resource pool and to render a static or randomized version into the LT3 environment. This will enable the LT3 as
-well as the LON-CAPA community to access the most recent version of each ConcepTest, as well as to track any changes which have been made by the individual authors. The common content pool
-allows us to
-carry out the proposed study with a minimum of logistical or technological challenges caused by the involvement of three different
-
-\subsection{Computer-Guided Group Formation} \label{groupform}
-The goal of computer-guided group formation is to generate student groups with differing initial responses. Group formation is limited by seating arrangements in a the lecture hall, unless time can be afforded for students to get up and walk around the lecture hall.
-
-As a first step, seating arrangements in the lecture hall need to be coded into a computer-readable format -- both the BQ and the LON-CAPA group have experience in this area. Then an algorithm needs to be developed to find the optimum configuration of groups of $N$ nearest neighboring students that maximizes initial dissent within the groups. Fig.~\ref{formation} is a mock-up of a possible configuration of two nearest neighbors within a lecture hall. We will implement a hybrid scheme, where internet based two-way communication, telling the individual student where to turn, will be combined with
-a projection of the map to the front of the class . This implementation is very flexible, since it can be used in almost any teaching environment independent of the technological resourcs available.
-
-
-\begin{figure}[t]
-\begin{center}
-\includegraphics[width=2.2in]{seatfig.eps}
-\caption{\small Computer-guided group formation.\label{formation}
-}
-\end{center}
-\end{figure}
-The system records group configurations and makes analyses of pre- and post-discussion responses within the groups possible.
-
-\subsection{Different Question Types}
-Key to deploying both different question types and randomizing questions (see following subsection~\ref{randomques}) are two-way interactive devices. At MSU, we will use a classroom set of 120 WiFi-enabled Dell Axim X3 PDAs for the study (purchased by MSU).
-
-
-\subsection{Randomized Questions}\label{randomques}
-The LON-CAPA system is already capable of randomizing questions with a wide range of options and also provides statistical
-tools for the de-randomized analysis of responses.
-For classroom use, specialized tools similar to the existing ILT functionality need to be developed to provide the
-instructor with a quick and comprehensive overview of response patterns of the more complex randomizing question types.
-
-\begin{figure}
-\includegraphics[width=3.5in]{dell.eps}
-\includegraphics[width=2.7in]{sharp2.eps}
-\caption{\small Rendering of a problem on PDA devices\label{pdaview}
-}
-\end{figure}
-
-
\section{Commodization Phase and Dissemination}
ubsection{Commodization Phase} \label{comphase}
While in the initial phases of the project, system functionality will be combined from the existing systems (subsection~\ref{existing}) by the PIs, for the dissemination of
--www1116445268--