[LON-CAPA-cvs] cvs: modules /gerd/roleclicker description.tex

[www]

This is a MIME encoded message

--www1115646440
Content-Type: text/plain

www		Mon May  9 09:47:20 2005 EDT

  Modified files:              
    /modules/gerd/roleclicker	description.tex 
  Log:
  Shuffling stuff around
  
  
--www1115646440
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20050509094720.txt"

Index: modules/gerd/roleclicker/description.tex
diff -u modules/gerd/roleclicker/description.tex:1.7 modules/gerd/roleclicker/description.tex:1.8
--- modules/gerd/roleclicker/description.tex:1.7	Sun May  8 22:10:53 2005
+++ modules/gerd/roleclicker/description.tex	Mon May  9 09:47:19 2005
@@ -73,7 +73,7 @@
 is entirely based on seating arrangements: ``turn to the student sitting next to you.'' Peer instruction works best if learners 
 need to convince each other and discuss the merrits of different possible solutions - which is not likely to happen naturally if all
 learners in the randomly formed group initially chose the same option. In computer-guided group formation, the computer will be forming the groups based on the initial learner responses, and ensure that within the constraints of the lecture hall seating arrangement, groups with a diversity of initial opinions are formed.  
-\item[Different Question Types] - in current practice, the questions presented to learners are mostly multiple-choice style. This is due to the limitation of current response mechanisms. More sophisticated response devices allow for the deployment of more sophisticated question types, such as image-response, mix-and-match, multiple-response multiple-choice and open-ended numerical/symbolic math questions.
+\item[Different Question Types] - in current practice, the questions presented to learners are mostly multiple-choice style. This is due to the limitation of current response mechanisms. More sophisticated response devices allow for the deployment of more sophisticated question types, such as image-response, mix-and-match, multiple-response multiple-choice and open-ended numerical/symbolic math questions. Of particular interest will be the incorporation of simulations into the classroom.
 \item[Randomized Questions] - in current practice, all students in a course are answering the exact same question. More sophisticated response devices allow for randomizing scenarios just enough such that students can discuss the same underlying principle, yet still need to draw their own conclusions and arrive at their own solutions.
 \end{description}
 
@@ -89,9 +89,7 @@
 \item Commoditizing and dissemination of successful techniques
 \end{enumerate}
 
-
-
-We will focus on physics content, where a broad research base on existing techniques already exists, and where research-based content for peer-teaching using current techniques is readily available. We will work with undergraduate students in large-enrollment introductory algebra- and calculus-based physics courses at a spectrum of institutions.
+We will focus on physics content, where a broad research base on existing techniques already exists, and where research-based content for peer-teaching using current techniques is readily available. We will work with undergraduate students in introductory algebra- and calculus-based physics courses at a spectrum of institutions.
 \subsection{Project Partners}
 \begin{itemize}
 \item Gerd Kortemeyer and Guy Albertelli at Michigan State University
@@ -99,7 +97,7 @@
 \item Bill Junkin at Erskine College
 \end{itemize}
 \subsection{Intellectual Merit}
-
+Peer-instruction has proven successful in outcome-oriented evaluations of techniques limited by currently widely available technology. This project will add process-oriented data to the research body around peer-instruction and study the effect of extensions to this technique, which we believe can significantly change both the process and the outcome of applying these techniques.
 \subsection{Broader Impact}
 Currently, every semester approximately 350,000 US students are taking introductory undergraduate physics courses similar to the ones under investigation in this project~\cite{aapt}. For many of these students, it is both their first and their last formal exposure to physics. Students will go into a large spectrum of careers, with or without an understanding of the basic concepts of the physical world.
 
@@ -113,248 +111,18 @@
 faculty. Faculty members at the over thirty currently participating LON-CAPA
 institutions will be able to profit from this project already during its
 progress.
-\section{Background}
-\subsection{Peer-Instruction}
-Since developing Peer-Instruction (PI), a collaborative and interactive teaching technique, in 1991, we
-have been 1) disseminating the technique, 2) gathering data on its effectiveness (see publication list
-below), and 3) developing web-based tools to help instructors implement the method. We have applied
-the method in both the calculus-based and the algebra-based introductory physics courses for non-majors
-at Harvard University. Instructors nationwide have adopted the method across a variety of disciplines and
-courses, including senior-level courses, at a large number of institutions nationwide. Substantial gains in
-student achievement when comparing courses taught using PI to those taught with
-traditional pedagogy
-have been documented. These gains have been determined by a number of measures,
-including student
-mastery of content \cite{mref1,mref2,mref3,mref4,mref5,mref6,mref7,mref8,mref9,mref10}.
-The trend in improving student understanding proves to be particularly
-beneficial to female students, whose performance increases substantially, when taught using this
-interactive method \cite{mref13}.
-The primary resource needed for teaching with PI is a supply of suitable ConcepTests (CTs) -
-questions that test students' understanding of the basic concepts covered \cite{mref11}. We have developed and
-refined over 1,000 CTs for use in introductory physics courses. These CTs are freely available to
-instructors through the ILT web site (detailed below), together with over 400 additional CTs that have
-been contributed by others. An indicator of the rapid spread of the method is the availability of books
-with ConcepTests for chemistry, astronomy, and calculus courses. We are currently in the process of
-adding this material to the ILT web site.
 
-Under NSF sponsorship, we developed Project Galileo4, a store of extensive resources for
-interactive learning pedagogies, targeting both large and small classroom teaching techniques, which are
-available to the entire teaching community. Using funds from a NSF Director's Distinguished Teaching
-Scholar Award, we created the Interactive Learning Toolkit, a learning management system that allows
-instructors to implement several proven innovative teaching techniques and to share and review materials
-they create for these techniques. The ILT is currently in use at a number of institutions nationwide,
-including Vanderbilt, University of Southern California, University of Massachusetts-Boston, Salem State
-College, Massachusetts Institute of Technology, Swarthmore College, with a student user base of several
-thousand students per semester.
-We also invested a great deal of effort disseminating our findings nationwide, as we feel that it is
-crucial to share the results of our research. In the last several years, Eric Mazur and other members of the
-group have given more than one hundred invited talks on PI in a variety of venues:
-\begin{itemize}
-\item Physics department colloquia at a wide range of institutions from large state universities to small
-liberal arts colleges and community colleges;
-\item Workshops for new faculty sponsored by the American Association of Physics Teachers and the NSFfunded
-Engineering Education Scholars program;
-\end{itemize}
-\subsection{Collaborative Learning}
-There is a significant body of literature concerning theories of and best practices for collaborative
-learning \cite{mref14}. In general, the motivation behind collaborative learning is to make students active
-participants in the learning process, assigning them more responsibility for their own education. Students
-explore concepts in depth as opposed to the "inch-deep, mile-wide" curriculum so common in
-traditionally taught classes. However, collaborative learning need not be at the cost of "coverage". By
-moving some of the non-interactive activities out of the classroom, it is possible to maintain coverage
-without sacrificing in depth discussion. Many researchers have come to the conclusion that promoting
-discussion between students in learning communities is the most important influence of the effectiveness
-of education \cite{mref14,mref15}. Moreover, the quality of this interaction between students is a crucial factor for
-determining the success of learning \cite{mref14}.
 
-Treisman's pioneering work on collaborative learning \cite{mref16,mref17,mref18} demonstrates that Asian students
-in college calculus at the University of California-Berkeley typically study in
-groups, help each other with
-homework and exam preparation, and often consult with the instructor for help with difficult assignments.
-These students are disproportionately successful in the course. In contrast, African-American students
-typically work alone, do not seek help outside of class from the instructor, and, on average, do not
-perform as well in the course. Treisman therefore established a program to foster collaborative work
-among African-American students, finding that it resulted in a dramatic improvement in the performance
-of these students in the course. Although the program does not require it, students in the collaborative
-program also tend to consult with the instructor outside of class more often.
-Dougherty et al. \cite{mref19} carried out a controlled study on educational effectiveness of three
-pedagogical strategies in different sections of an introductory college-level chemistry course: standard
-lecture format, unstructured cooperative interaction, and formal collaborative exercises. The authors
-found that the unstructured cooperative environment is significantly more effective than the control, and
-the formal structured environment is significantly more effective than the unstructured. Kovac \cite{mref20}
-conducted a similar study in a general chemistry course, employing many of the features we use in our
-physics courses at Harvard, such as teaching using CTs, and cooperative learning workshops/tutorials.
-Results from this study, as with Dougherty above indicate that active learning methods appear to produce
-a better learning environment, leading to an increase in student satisfaction and better academic
-performance.
-Peer Instruction (PI) is a promising method for affecting fundamental, systemic
-improvement in
-science education \cite{mref21,mref22,mref23}. The goal of PI is to promote student interaction in classes (including large
-classes) and to focus students' attention on underlying concepts. The class is broken up into a number of
-brief discussions of the key points of the material. Each of these discussions includes one or more CTs -
-a short conceptual questions that challenge students to put the material at hand into practice. Students are
-given about a minute to formulate and record an individual answer to the question posed and are then
-asked to try to convince neighboring students of the correctness of their own answers. After a few minutes
-of peer-to-peer discussion, students record their possibly revised answers, usually followed by a
-clarification by the instructor reinforcing the main concept.
-PI demands that students think critically about the material and participate actively in the learning
-process; in addition, it uncovers student misunderstandings in real time so that they can be identified and
-corrected at once. PI is also particularly efficient because it helps those who
-get the answer right as well
-as those who get it wrong. Students answering correctly improve their own understanding by explaining
-CTs to others (consistent with research that shows high-ability students benefit from collaboration
-\cite{mref25,mref26}), and students answering incorrectly benefit from individualized explanations and the opportunity
-to ask follow-up questions of their classmates.
-Our ten years of experience with PI, as well as feedback from about 400 other instructors who
-have used PI \cite{mref27}, indicate that it is a successful way to actively engage students in large classes.
-Moreover, actively engaging students during class with a method such as PI leads to significant gains in
-conceptual understanding, as measured with standard conceptual instruments. Students in our calculusbased
-introductory physics course achieve Force Concept Inventory gains that are roughly twice those of
-students in the same course taught traditionally, a level of improvement typical of a variety of interactive
-engagement strategies in physics \cite{mref28}. Students also show comparable or improved quantitative problemsolving
-skills, despite a reduced emphasis on problems in class \cite{mref11,mref12}.
-Research on collaborative education nearly universally indicates that collaborative work is more
-effective than passive learning. Our experiences with PI, as well as those of many others, who have
-responded to our survey, show PI to be an effective collaborative approach to learning.
- 
-\subsection{Interactive Learning Toolkit}
-Two years ago, we published the result of an online survey of PI users in diverse settings. \cite{mref27} This
-survey alerted us to some key areas that instructors found challenging, including the time taken to
-structure and organize teaching with PI efficiently, the lack of teaching materials (specifically CTs), use
-of class time to encourage student-instructor interaction and increasing student interaction and
-participation in the course. As a result, we created and developed a learning management system, which
-not only supports and improves the teaching and learning experience, but also targets these key problem
-areas. The resulting ILT is an open-source project based on open standards and is currently in use at a
-number of collaborating institutions nationwide. With help from current users, we are continuing to
-develop content and tools for interactive learning pedagogies.
-Our progress to date has involved the development of the following features:
-\begin{itemize}
-\item Course web site creation. We created a set of tools that allow an instructor to structure and create (or
-search from our ever-growing database of questions) content for their class, make that content
-available for students, and then analyze the feedback from the students.
-\item Reading assignments. In order to "free-up" precious class time for more interactive activities, we
-developed an on-line pre-class reading assignment tool.
-\item Face book. To help improve the personalized interaction between the student and instructor, we
-developed a "face book" in ILT. What this means is that anywhere a student's name appears in ILT, it
-links it to their picture, and also to a portal page showing their progress in all aspects of the course.
-This novel tool helps the instructor to become more familiar with each student,
-helping improve
-individual interaction and to quickly identify students who might be struggling
-in the course. This is
-of particular pedagogical value in a large class, where students normally remain anonymous.
-\item Asynchronous student-instructor interaction. To increase student-instructor interaction, the toolkit
-incorporates a number of innovative communication features. It incorporates an interface that enables
-the instructor to process student responses to online assignments in a large class and quickly and
-efficiently and to respond to students who need help. ILT speeds up reviewing the students' responses
-and answering them quickly by storing answers for re-use, by identifying similar student responses
-and making it possible to quickly send individualized answers to these students
-via e-mail. A similar
-mechanism is also available to the instructor to handle email from the students
-enrolled in the course.
-\item Equal attention to every student. We built in a mechanism to encourage instructors to give all students
-equal attention. The system monitors instructor-student interactions and sorts the list of students in
-such a way that those whom the instructor has interacted with least in the term
-are always listed at the
-top. This mechanism encourages an instructor to spend equal time interacting with every student.
-\item Re-use of material. Once an instructor has created a course online, the ILT has a cloning feature that
-allows the instructor to clone the course and re-use it in whole, or adapt specific content within it. A
-calendar-based scheduling feature allows the quick development of a new course schedule,
-incorporating material used in a previous iteration of the course.
-\item Online assessment. The ILT also provides an extensive assessment feature that allows instructors to
-develop and use their own assessment instruments, or use one of an available assortment of
-standardized tests.
-\item Sharing. Finally, and most importantly, ILT provides a location to "warehouse" course content so it
-can be shared by the entire community of instructors. An simple rights management system allows
-instructors to either maintain their copyright or to place their material in the public domain.
-\item Personal Response System (PRS). We have also begun to incorporate support
-for gathering student
-responses to CTs using infrared PRS transmitters. The resulting data can be uploaded to the database
-to their corresponding CT, allowing instructors to review the efficacy of a particular CT after class. In
-addition the CT data is also connected to the students' records, so that the instructor can review an
-individual student performance.
-\end{itemize}
- 
-
-
-\subsection{The Learning{\it Online} Network with CAPA}\label{loncapa}
-The Learning{\it Online} Network with Computer-Assisted Personalized Approach ({\tt http://www.lon-capa.org/}) is a distributed learning content management, course management, and assessment system, and also the model system of the current NSF-ITR grant, see Sect.~\ref{results}. 
-
-LON-CAPA's core development group is located at MSU, and in addition to faculty members, has a staff of three fulltime programmers, two user support staff, one technician, one graduate student, and one project coordinator. The LON-CAPA group also offers training and support for adopters of the system.
-
-LON-CAPA is open-source (GNU General Public License) freeware, there are no licensing costs associated. Both aspects are important for the success of this project: the open-source nature of the system allows researchers to modify and adapt the system in order to address research needs, and the freeware character allows easier dissemination of results, in particular adaptation and implementation at other universities.
-\subsubsection{Shared Distributed Content Repository}
-LON-CAPA is designed around the concept of easy sharing and re-use of learning resources. 
-
-In LON-CAPA, the underlying distributed multimedia content repository spans across all of the currently over 30 participating institutions, and currently contains over 60,000 learning content resources, including more than 18,000 personalized homework problems. Disciplines include astronomy, biology, business, chemistry, civil engineering, computer science, family and child ecology, geology, human food and nutrition, human medicine, mathematics, medical technology, physics, and psychology. Any content material contributed to the pool is immediately available and ready-to-use within the system at all participating sites, thus facilitating dissemination of curricular development efforts (Sect.~\ref{matdev}). A large fraction of these resources are also available through the gateway to the National Science Digital Library (NSDL).
-
-Navigation through selected resources is provided by an internal sequencing tool, which allows assembling, re-using, and re-purposing content at different levels of granularity (pages, lessons, modules, chapters, etc) --- each content assembly becomes a new resource in the system.
-
-
-The network provides constant assessment of the resource quality through objective and subjective dynamic metadata. Selection of a learning resource by instructors at other institutions while constructing a learning module does both establish a de-facto peer-review mechanism and provide additional context information for each resource. In addition, access statistics are being kept, and learners can put evaluation information on each resource.
-
-In addition to faculty-provided content, the problem supplements to a number of commercial textbooks are available in LON-CAPA format.
-LON-CAPA provides highly customizable access control for such resources, and has a built-in key mechanism to charge for content access. 
-
-
-\subsubsection{Formative and Summative Assessment Capabilities}
-LON-CAPA started in 1992 as a system to give personalized homework to students in introductory physics courses.  ``Personalized" means that each student sees a different version of the same computer-generated problem: different numbers, choices, graphs, images, simulation parameters, etc, Fig.~\ref{twoproblems}.
-
-\begin{figure}
-\includegraphics[width=6.5in]{atwood.eps}
-\caption{Web-rendering of the same LON-CAPA problem for two different students.\label{twoproblems}
-}
-\end{figure}
-
-\begin{figure}
-\includegraphics[width=3.5in]{dell.eps}
-\includegraphics[width=2.7in]{sharp2.eps}
-\caption{Rendering of a problem on PDA devices\label{pdaview}
-}
-\end{figure}
-In the context of this project, this feature is important in two aspects:
-\begin{itemize}
-\item results are not tainted by students simply exchanging the answers, i.e., each student in the end has to work out his or her own answers
-\item as a result, lively discussions take place, both online and in the helproom --- both of which will be analyzed in this project, see Sect.~\ref{discussion}
-\end{itemize}
-
-Students are generally given immediate feedback on the correctness of their solutions, and in some cases additional help. They are usually granted multiple attempts to get a problem correct. This allows the instructor to follow a learner's thought process, both through statistical analysis (see~\ref{anatool}) and data-mining approaches.
-
-The system also allows for free-form essay-type answers, which are however graded by humans with the assistance of the system (keyword-highlighting, plagiarism-checks, etc).
-\subsubsection{Course Management}
-Over the years, the system added a learning content management system and standard course management features, such as communications, gradebook, etc., which are comparable to commercial course management systems, such as BlackBoard, WebCT, or ANGEL. See 
-Refs.~\cite{features,edutools} for an overview of features, and comparisons to other systems.
-
-In addition to standard features, the LON-CAPA delivery and course management layer is designed around STEM education, for example: support for mathematical typesetting throughout (\LaTeX\ inside of XML) Ð-- formulas are rendered on-the-fly, and can be algorithmically modified through the use of variables inside formulas; integrated GNUplot support, such that graphs can be rendered on-the-fly, and allowing additional layered labeling of graphs and images; support for multi-dimensional symbolic math answers; and full support of physical units.
-
-\begin{figure}
-\includegraphics[width=6.5in]{problemview}
-\caption{Example an individual student view for problem analysis.\label{problemview}}
-\end{figure}
-
-\subsubsection{Analysis Capabilities}\label{anatool}
-LON-CAPA allows instructors to analyze student submissions both for individual students (Fig.~\ref{problemview}) and across the course (Fig.~\ref{problemanalysis}).
-
-For example, Fig.~\ref{problemview} indicates that in the presence of a medium between the capacitor plates, the student was convinced that the force would increase, but also that this statement was the one he was most unsure about: His first answer was that the force would double; no additional feedback except ``incorrect" was provided by the system. In his next attempt, he would change his answer on only this one statement (indicating that he was convinced of his other answers) to ``four times the force" --- however, only ten seconds passed between the attempts, showing that he was merely guessing by which factor the force increased. The graphs on the right of Fig.~\ref{problemanalysis} show which statements were answered correctly course-wide on the first and on the second attempt, respectively, the graphs on the right which other options the students chose if the statement was answered incorrectly. Clearly, students have the most difficulty with the concept of how a medium acts inside a capacitor, with the absolute majority believing the force would increase, and only about 20\% of the students believing the medium had no influence.
-
-\begin{figure}
-\begin{center}
-\includegraphics[width=5in]{problemanalysis.eps}
-\end{center}
-\caption{Example of a course-wide problem analysis for the problem Fig.~\ref{problemview}.\label{problemanalysis}}
-\end{figure}
+\section{Methodology}\label{method}
+The effectiveness of the extensions to peer-instruction will be evaluated by both with focus on process and on learning outcomes.
+\subsection{Process-Oriented Evaluation}
+The process-oriented evaluation will focus on the actual discussion process. Since currently no baseline data exists for this study, we will assess the quality of student discussion both
+before and after the introduction of extensions to the current peer-instruction technique.
 
+The PIs of this project have experience analysing student discussions from a prior project which examined asynchronous online student discussions around different types of online homework problems.
+After categorization of both the problem types and the discussion contributions, significant differences in the student discussion behaviour around different problem types could be extracted.
 
-\subsection{Courses}\label{coursesdesc}
-The project will be carried out  in the two-semester LBS course sequence LBS 271/272, ``Calculus-Based Introductory Physics I/II." These second-year non-major three-credit courses have a Calculus pre-requisite, and traditionally an enrollment of over 200 students. 
-
-Starting Fall 2004, the course will be taught with less total lecturing time, where the third classroom hour will be used for peer-teaching~\cite{mazur} and more frequent quizzes in place of the midterm exams.
-
-Two separate, but associated one-credit laboratory courses (LBS 271L/272L) are required, which most but not all students choose to take simultaneously. Faculty and teaching assistants are frequently assuming shared responsibilities between the lecture and lab courses, with a combined staff of two faculty members and six undergraduate student assistants. The latter are responsible for particular recitation and lab sections, and will be involved in this research project (see Sect.~\ref{undergrad}). Within the duration of this project, the lecture and lab courses might be combined to provide greater coherence between these two venues.
-
-Students in these courses are currently solving approximately 200 online homework problems each semester, most of which currently are of the conventional type.
-\section{\label{sec:method}Methodology}
-
-\subsection{\label{subsec:problemcat}Problem Classification}
+\subsubsection{\label{subsec:problemcat}Problem Classification}
 Kashy~\cite{kashyd01} showed that student mastery of different types of homework problems correlates differently with the students' performance on final exams --- 
 with multiple-choice non-numerical problems having the lowest correlation, and numerical/mathematical problems that require a translation of representation having the highest.
 Steinberg~\cite{steinberg} also analyzed student performance on multiple-choice diagnostics and open-ended exam problems, and found that while those correlate in general, for certain students
@@ -420,16 +188,16 @@
 
 In addition, for every question, its
 difficulty index was computed according to the formula
-\begin{equation*}\label{eqn:diffidx}
+\begin{equation}\label{eqn:diffidx}
 \mbox{Difficulty Index}=10\left(1-\frac{N_{\mbox{correct}}}{N_{\mbox{attempts}}}\right)
-\end{equation*}
+\end{equation}
 where $N_{\mbox{correct}}$ is the total number of correct solution in the course, and $N_{\mbox{attempts}}$ is the total number of
 correct and incorrect solution submissions (the system allows multiple attempts to arrive at the correct solution, see 
 subsection~\ref{subsec:system}). If all submissions were correct, meaning, every student would have solved the problem
 correctly on the first attempt, the difficulty index would be 0. If none of the submissions were correct, the index would be 10.
 
 
-\subsection{\label{subsec:disccat}Discussion Classification}
+\subsubsection{\label{subsec:disccat}Discussion Classification}
 Student discussion entries were classified into three types and four features. The four types are
 \begin{description}
 \item[Emotional] - discussion contributions were classified as ``emotional" if they mostly communicated opinions,
@@ -452,7 +220,7 @@
 Table~\ref{table:examples} shows examples of contributions and their classification.
 \begin{table*}
 \caption{Examples of discussion contribution types and features.\label{table:examples}} 
-\begin{ruledtabular}
+
 \begin{tabular}{l|p{3.9cm}|p{3.9cm}|p{3.9cm}|p{3.9cm}}
 &Unrelated&Solution&Math&Physics\\\hline
 Emotional&
@@ -491,7 +259,6 @@
 &
 I have the correct answer, but I don't understand why it is correct. Why would there be an acceleration at the ball's highest point? Why wouldn't it be zero?
 \end{tabular}
-\end{ruledtabular}
 \end{table*}
 Discussion contributions were always classified as a whole, and since they were fairly short, they mostly fell clearly into one of the classes. If a longer contribution had aspects of more than one class, it was characterized by
 the class that its majority fell into. Discussion contributions by teaching assistants and instructors were not 
@@ -502,7 +269,6 @@
 subsection~\ref{subsec:disccat}. The columns denote the different discussion types and subtypes, while the 
 rows denote the 
 features.\label{table:disccat}}
-\begin{ruledtabular}
 \begin{tabular}{lcccccccc|l}
 &\multicolumn{2}{c}{Emotional}
 &\multicolumn{2}{c}{Surface}
@@ -516,7 +282,6 @@
  Physics  &     &   14&   85&   81&  170&  190&  100&  126&766\\\hline
           &  351&  259&  745&  459&  596&  733&  116&  135&3394
 \end{tabular}
-\end{ruledtabular}
 \end{table}
 
 
@@ -684,6 +449,261 @@
 homework problems~\cite{lin}. 
 
 
+
+
+
+\subsection{Outcome-Oriented Evaluation}
+ 
+
+
+%
+%
+% Copy-Paste
+%
+
+
+
+\section{Background}
+\subsection{Peer-Instruction}
+Since developing Peer-Instruction (PI), a collaborative and interactive teaching technique, in 1991, we
+have been 1) disseminating the technique, 2) gathering data on its effectiveness (see publication list
+below), and 3) developing web-based tools to help instructors implement the method. We have applied
+the method in both the calculus-based and the algebra-based introductory physics courses for non-majors
+at Harvard University. Instructors nationwide have adopted the method across a variety of disciplines and
+courses, including senior-level courses, at a large number of institutions nationwide. Substantial gains in
+student achievement when comparing courses taught using PI to those taught with
+traditional pedagogy
+have been documented. These gains have been determined by a number of measures,
+including student
+mastery of content \cite{mref1,mref2,mref3,mref4,mref5,mref6,mref7,mref8,mref9,mref10}.
+The trend in improving student understanding proves to be particularly
+beneficial to female students, whose performance increases substantially, when taught using this
+interactive method \cite{mref13}.
+The primary resource needed for teaching with PI is a supply of suitable ConcepTests (CTs) -
+questions that test students' understanding of the basic concepts covered \cite{mref11}. We have developed and
+refined over 1,000 CTs for use in introductory physics courses. These CTs are freely available to
+instructors through the ILT web site (detailed below), together with over 400 additional CTs that have
+been contributed by others. An indicator of the rapid spread of the method is the availability of books
+with ConcepTests for chemistry, astronomy, and calculus courses. We are currently in the process of
+adding this material to the ILT web site.
+
+Under NSF sponsorship, we developed Project Galileo4, a store of extensive resources for
+interactive learning pedagogies, targeting both large and small classroom teaching techniques, which are
+available to the entire teaching community. Using funds from a NSF Director's Distinguished Teaching
+Scholar Award, we created the Interactive Learning Toolkit, a learning management system that allows
+instructors to implement several proven innovative teaching techniques and to share and review materials
+they create for these techniques. The ILT is currently in use at a number of institutions nationwide,
+including Vanderbilt, University of Southern California, University of Massachusetts-Boston, Salem State
+College, Massachusetts Institute of Technology, Swarthmore College, with a student user base of several
+thousand students per semester.
+We also invested a great deal of effort disseminating our findings nationwide, as we feel that it is
+crucial to share the results of our research. In the last several years, Eric Mazur and other members of the
+group have given more than one hundred invited talks on PI in a variety of venues:
+\begin{itemize}
+\item Physics department colloquia at a wide range of institutions from large state universities to small
+liberal arts colleges and community colleges;
+\item Workshops for new faculty sponsored by the American Association of Physics Teachers and the NSFfunded
+Engineering Education Scholars program;
+\end{itemize}
+\subsection{Collaborative Learning}
+There is a significant body of literature concerning theories of and best practices for collaborative
+learning \cite{mref14}. In general, the motivation behind collaborative learning is to make students active
+participants in the learning process, assigning them more responsibility for their own education. Students
+explore concepts in depth as opposed to the "inch-deep, mile-wide" curriculum so common in
+traditionally taught classes. However, collaborative learning need not be at the cost of "coverage". By
+moving some of the non-interactive activities out of the classroom, it is possible to maintain coverage
+without sacrificing in depth discussion. Many researchers have come to the conclusion that promoting
+discussion between students in learning communities is the most important influence of the effectiveness
+of education \cite{mref14,mref15}. Moreover, the quality of this interaction between students is a crucial factor for
+determining the success of learning \cite{mref14}.
+
+Treisman's pioneering work on collaborative learning \cite{mref16,mref17,mref18} demonstrates that Asian students
+in college calculus at the University of California-Berkeley typically study in
+groups, help each other with
+homework and exam preparation, and often consult with the instructor for help with difficult assignments.
+These students are disproportionately successful in the course. In contrast, African-American students
+typically work alone, do not seek help outside of class from the instructor, and, on average, do not
+perform as well in the course. Treisman therefore established a program to foster collaborative work
+among African-American students, finding that it resulted in a dramatic improvement in the performance
+of these students in the course. Although the program does not require it, students in the collaborative
+program also tend to consult with the instructor outside of class more often.
+Dougherty et al. \cite{mref19} carried out a controlled study on educational effectiveness of three
+pedagogical strategies in different sections of an introductory college-level chemistry course: standard
+lecture format, unstructured cooperative interaction, and formal collaborative exercises. The authors
+found that the unstructured cooperative environment is significantly more effective than the control, and
+the formal structured environment is significantly more effective than the unstructured. Kovac \cite{mref20}
+conducted a similar study in a general chemistry course, employing many of the features we use in our
+physics courses at Harvard, such as teaching using CTs, and cooperative learning workshops/tutorials.
+Results from this study, as with Dougherty above indicate that active learning methods appear to produce
+a better learning environment, leading to an increase in student satisfaction and better academic
+performance.
+Peer Instruction (PI) is a promising method for affecting fundamental, systemic
+improvement in
+science education \cite{mref21,mref22,mref23}. The goal of PI is to promote student interaction in classes (including large
+classes) and to focus students' attention on underlying concepts. The class is broken up into a number of
+brief discussions of the key points of the material. Each of these discussions includes one or more CTs -
+a short conceptual questions that challenge students to put the material at hand into practice. Students are
+given about a minute to formulate and record an individual answer to the question posed and are then
+asked to try to convince neighboring students of the correctness of their own answers. After a few minutes
+of peer-to-peer discussion, students record their possibly revised answers, usually followed by a
+clarification by the instructor reinforcing the main concept.
+PI demands that students think critically about the material and participate actively in the learning
+process; in addition, it uncovers student misunderstandings in real time so that they can be identified and
+corrected at once. PI is also particularly efficient because it helps those who
+get the answer right as well
+as those who get it wrong. Students answering correctly improve their own understanding by explaining
+CTs to others (consistent with research that shows high-ability students benefit from collaboration
+\cite{mref25,mref26}), and students answering incorrectly benefit from individualized explanations and the opportunity
+to ask follow-up questions of their classmates.
+Our ten years of experience with PI, as well as feedback from about 400 other instructors who
+have used PI \cite{mref27}, indicate that it is a successful way to actively engage students in large classes.
+Moreover, actively engaging students during class with a method such as PI leads to significant gains in
+conceptual understanding, as measured with standard conceptual instruments. Students in our calculusbased
+introductory physics course achieve Force Concept Inventory gains that are roughly twice those of
+students in the same course taught traditionally, a level of improvement typical of a variety of interactive
+engagement strategies in physics \cite{mref28}. Students also show comparable or improved quantitative problemsolving
+skills, despite a reduced emphasis on problems in class \cite{mref11,mref12}.
+Research on collaborative education nearly universally indicates that collaborative work is more
+effective than passive learning. Our experiences with PI, as well as those of many others, who have
+responded to our survey, show PI to be an effective collaborative approach to learning.
+ 
+\subsection{Interactive Learning Toolkit}
+Two years ago, we published the result of an online survey of PI users in diverse settings. \cite{mref27} This
+survey alerted us to some key areas that instructors found challenging, including the time taken to
+structure and organize teaching with PI efficiently, the lack of teaching materials (specifically CTs), use
+of class time to encourage student-instructor interaction and increasing student interaction and
+participation in the course. As a result, we created and developed a learning management system, which
+not only supports and improves the teaching and learning experience, but also targets these key problem
+areas. The resulting ILT is an open-source project based on open standards and is currently in use at a
+number of collaborating institutions nationwide. With help from current users, we are continuing to
+develop content and tools for interactive learning pedagogies.
+Our progress to date has involved the development of the following features:
+\begin{itemize}
+\item Course web site creation. We created a set of tools that allow an instructor to structure and create (or
+search from our ever-growing database of questions) content for their class, make that content
+available for students, and then analyze the feedback from the students.
+\item Reading assignments. In order to "free-up" precious class time for more interactive activities, we
+developed an on-line pre-class reading assignment tool.
+\item Face book. To help improve the personalized interaction between the student and instructor, we
+developed a "face book" in ILT. What this means is that anywhere a student's name appears in ILT, it
+links it to their picture, and also to a portal page showing their progress in all aspects of the course.
+This novel tool helps the instructor to become more familiar with each student,
+helping improve
+individual interaction and to quickly identify students who might be struggling
+in the course. This is
+of particular pedagogical value in a large class, where students normally remain anonymous.
+\item Asynchronous student-instructor interaction. To increase student-instructor interaction, the toolkit
+incorporates a number of innovative communication features. It incorporates an interface that enables
+the instructor to process student responses to online assignments in a large class and quickly and
+efficiently and to respond to students who need help. ILT speeds up reviewing the students' responses
+and answering them quickly by storing answers for re-use, by identifying similar student responses
+and making it possible to quickly send individualized answers to these students
+via e-mail. A similar
+mechanism is also available to the instructor to handle email from the students
+enrolled in the course.
+\item Equal attention to every student. We built in a mechanism to encourage instructors to give all students
+equal attention. The system monitors instructor-student interactions and sorts the list of students in
+such a way that those whom the instructor has interacted with least in the term
+are always listed at the
+top. This mechanism encourages an instructor to spend equal time interacting with every student.
+\item Re-use of material. Once an instructor has created a course online, the ILT has a cloning feature that
+allows the instructor to clone the course and re-use it in whole, or adapt specific content within it. A
+calendar-based scheduling feature allows the quick development of a new course schedule,
+incorporating material used in a previous iteration of the course.
+\item Online assessment. The ILT also provides an extensive assessment feature that allows instructors to
+develop and use their own assessment instruments, or use one of an available assortment of
+standardized tests.
+\item Sharing. Finally, and most importantly, ILT provides a location to "warehouse" course content so it
+can be shared by the entire community of instructors. An simple rights management system allows
+instructors to either maintain their copyright or to place their material in the public domain.
+\item Personal Response System (PRS). We have also begun to incorporate support
+for gathering student
+responses to CTs using infrared PRS transmitters. The resulting data can be uploaded to the database
+to their corresponding CT, allowing instructors to review the efficacy of a particular CT after class. In
+addition the CT data is also connected to the students' records, so that the instructor can review an
+individual student performance.
+\end{itemize}
+ 
+
+
+\subsection{The Learning{\it Online} Network with CAPA}\label{loncapa}
+The Learning{\it Online} Network with Computer-Assisted Personalized Approach ({\tt http://www.lon-capa.org/}) is a distributed learning content management, course management, and assessment system, and also the model system of the current NSF-ITR grant, see Sect.~\ref{results}. 
+
+LON-CAPA's core development group is located at MSU, and in addition to faculty members, has a staff of three fulltime programmers, two user support staff, one technician, one graduate student, and one project coordinator. The LON-CAPA group also offers training and support for adopters of the system.
+
+LON-CAPA is open-source (GNU General Public License) freeware, there are no licensing costs associated. Both aspects are important for the success of this project: the open-source nature of the system allows researchers to modify and adapt the system in order to address research needs, and the freeware character allows easier dissemination of results, in particular adaptation and implementation at other universities.
+\subsubsection{Shared Distributed Content Repository}
+LON-CAPA is designed around the concept of easy sharing and re-use of learning resources. 
+
+In LON-CAPA, the underlying distributed multimedia content repository spans across all of the currently over 30 participating institutions, and currently contains over 60,000 learning content resources, including more than 18,000 personalized homework problems. Disciplines include astronomy, biology, business, chemistry, civil engineering, computer science, family and child ecology, geology, human food and nutrition, human medicine, mathematics, medical technology, physics, and psychology. Any content material contributed to the pool is immediately available and ready-to-use within the system at all participating sites, thus facilitating dissemination of curricular development efforts (Sect.~\ref{matdev}). A large fraction of these resources are also available through the gateway to the National Science Digital Library (NSDL).
+
+Navigation through selected resources is provided by an internal sequencing tool, which allows assembling, re-using, and re-purposing content at different levels of granularity (pages, lessons, modules, chapters, etc) --- each content assembly becomes a new resource in the system.
+
+
+The network provides constant assessment of the resource quality through objective and subjective dynamic metadata. Selection of a learning resource by instructors at other institutions while constructing a learning module does both establish a de-facto peer-review mechanism and provide additional context information for each resource. In addition, access statistics are being kept, and learners can put evaluation information on each resource.
+
+In addition to faculty-provided content, the problem supplements to a number of commercial textbooks are available in LON-CAPA format.
+LON-CAPA provides highly customizable access control for such resources, and has a built-in key mechanism to charge for content access. 
+
+
+\subsubsection{Formative and Summative Assessment Capabilities}
+LON-CAPA started in 1992 as a system to give personalized homework to students in introductory physics courses.  ``Personalized" means that each student sees a different version of the same computer-generated problem: different numbers, choices, graphs, images, simulation parameters, etc, Fig.~\ref{twoproblems}.
+
+\begin{figure}
+\includegraphics[width=6.5in]{atwood.eps}
+\caption{Web-rendering of the same LON-CAPA problem for two different students.\label{twoproblems}
+}
+\end{figure}
+
+\begin{figure}
+\includegraphics[width=3.5in]{dell.eps}
+\includegraphics[width=2.7in]{sharp2.eps}
+\caption{Rendering of a problem on PDA devices\label{pdaview}
+}
+\end{figure}
+In the context of this project, this feature is important in two aspects:
+\begin{itemize}
+\item results are not tainted by students simply exchanging the answers, i.e., each student in the end has to work out his or her own answers
+\item as a result, lively discussions take place, both online and in the helproom --- both of which will be analyzed in this project, see Sect.~\ref{discussion}
+\end{itemize}
+
+Students are generally given immediate feedback on the correctness of their solutions, and in some cases additional help. They are usually granted multiple attempts to get a problem correct. This allows the instructor to follow a learner's thought process, both through statistical analysis (see~\ref{anatool}) and data-mining approaches.
+
+The system also allows for free-form essay-type answers, which are however graded by humans with the assistance of the system (keyword-highlighting, plagiarism-checks, etc).
+\subsubsection{Course Management}
+Over the years, the system added a learning content management system and standard course management features, such as communications, gradebook, etc., which are comparable to commercial course management systems, such as BlackBoard, WebCT, or ANGEL. See 
+Refs.~\cite{features,edutools} for an overview of features, and comparisons to other systems.
+
+In addition to standard features, the LON-CAPA delivery and course management layer is designed around STEM education, for example: support for mathematical typesetting throughout (\LaTeX\ inside of XML) Ð-- formulas are rendered on-the-fly, and can be algorithmically modified through the use of variables inside formulas; integrated GNUplot support, such that graphs can be rendered on-the-fly, and allowing additional layered labeling of graphs and images; support for multi-dimensional symbolic math answers; and full support of physical units.
+
+\begin{figure}
+\includegraphics[width=6.5in]{problemview}
+\caption{Example an individual student view for problem analysis.\label{problemview}}
+\end{figure}
+
+\subsubsection{Analysis Capabilities}\label{anatool}
+LON-CAPA allows instructors to analyze student submissions both for individual students (Fig.~\ref{problemview}) and across the course (Fig.~\ref{problemanalysis}).
+
+For example, Fig.~\ref{problemview} indicates that in the presence of a medium between the capacitor plates, the student was convinced that the force would increase, but also that this statement was the one he was most unsure about: His first answer was that the force would double; no additional feedback except ``incorrect" was provided by the system. In his next attempt, he would change his answer on only this one statement (indicating that he was convinced of his other answers) to ``four times the force" --- however, only ten seconds passed between the attempts, showing that he was merely guessing by which factor the force increased. The graphs on the right of Fig.~\ref{problemanalysis} show which statements were answered correctly course-wide on the first and on the second attempt, respectively, the graphs on the right which other options the students chose if the statement was answered incorrectly. Clearly, students have the most difficulty with the concept of how a medium acts inside a capacitor, with the absolute majority believing the force would increase, and only about 20\% of the students believing the medium had no influence.
+
+\begin{figure}
+\begin{center}
+\includegraphics[width=5in]{problemanalysis.eps}
+\end{center}
+\caption{Example of a course-wide problem analysis for the problem Fig.~\ref{problemview}.\label{problemanalysis}}
+\end{figure}
+
+
+\subsection{Courses}\label{coursesdesc}
+The project will be carried out  in the two-semester LBS course sequence LBS 271/272, ``Calculus-Based Introductory Physics I/II." These second-year non-major three-credit courses have a Calculus pre-requisite, and traditionally an enrollment of over 200 students. 
+
+Starting Fall 2004, the course will be taught with less total lecturing time, where the third classroom hour will be used for peer-teaching~\cite{mazur} and more frequent quizzes in place of the midterm exams.
+
+Two separate, but associated one-credit laboratory courses (LBS 271L/272L) are required, which most but not all students choose to take simultaneously. Faculty and teaching assistants are frequently assuming shared responsibilities between the lecture and lab courses, with a combined staff of two faculty members and six undergraduate student assistants. The latter are responsible for particular recitation and lab sections, and will be involved in this research project (see Sect.~\ref{undergrad}). Within the duration of this project, the lecture and lab courses might be combined to provide greater coherence between these two venues.
+
+Students in these courses are currently solving approximately 200 online homework problems each semester, most of which currently are of the conventional type.
+\section{\label{sec:method}Methodology}
+
 \section{Research Methodology}\label{analysis}
 \subsection{Establishment of Initial Conditions}
 Many educational studies result in ``no-significant-difference"~\cite{russell}, and particularly the study of question type effectiveness (Sect.~\ref{effect}) may well yield the same result. Many variables may influence the impact of a particular sample of representative problems of a particular type for a particular learner, and it is imperative to understand as much of the ``initial conditions" as possible, since the validity of the hypothesis may depend on them.

--www1115646440--