[LON-CAPA-cvs] cvs: modules /gerd/roleclicker description.tex
www
lon-capa-cvs@mail.lon-capa.org
Wed, 18 May 2005 15:14:28 -0000
This is a MIME encoded message
--www1116429268
Content-Type: text/plain
www Wed May 18 11:14:28 2005 EDT
Modified files:
/modules/gerd/roleclicker description.tex
Log:
Moving existing infrastructure and previous results up front.
--www1116429268
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20050518111428.txt"
Index: modules/gerd/roleclicker/description.tex
diff -u modules/gerd/roleclicker/description.tex:1.56 modules/gerd/roleclicker/description.tex:1.57
--- modules/gerd/roleclicker/description.tex:1.56 Wed May 18 10:16:19 2005
+++ modules/gerd/roleclicker/description.tex Wed May 18 11:14:26 2005
@@ -1,5 +1,4 @@
-
-\documentclass[10pt]{article}
+\documentclass[11pt]{article}
\newif\ifpdf
\ifx\pdfoutput\undefined
\pdffalse % we are not running PDFLaTeX
@@ -113,26 +112,132 @@
The tools developed in the proposed project are flexible and can be used nationwide by instiutions with very different resources, including community colleges and high schools.
Both the tool and any developed, implemented, and adapted materials will be readily available for physics faculty.
-\section{Results from Prior NSF Support}\label{results}
-Gerd Kortemeyer is PI on the current NSF-ITR grant Investigation of a Model for Online Resource Creation and Sharing in Educational Settings (\#0085921, \$2,055,000, September 15, 2000 through July 31, 2005), which uses LON-CAPA as its model system.
+\section{Relevant Results from Prior NSF Support}\label{results}
+\subsection{MSU Group}\label{priormsu}
+Gerd Kortemeyer is PI on the current NSF-ITR grant Investigation of a Model for Online Resource Creation and Sharing in Educational Settings (\#0085921, \$2,055,000, September 15, 2000 through July 31, 2005). The project
+developed the cross-institutional learning content management system Learning{\it Online} Network with Computer Assisted Personalized Approach (LON-CAPA) and researched methods to assess the educational impact of
+content resources and representations within its shared content pool.
+
+LON-CAPA is open-source (GNU General Public License) freeware, there are no licensing costs associated.
+Both aspects are important for the success of this project: the open-source nature of the system allows researchers to modify and adapt the system in order to address research needs, and the freeware character allows
+easier dissemination of results, in particular adaptation and implementation at other universities.
+
+LON-CAPA is designed around the concept of easy sharing and re-use of learning resources, as well as advanced assessment capabilities in science courses.
+The system started in 1992 as a system to give personalized homework to students in introductory physics courses. ``Personalized" means that each student sees a different version of the same
+computer-generated problem: different numbers, choices, graphs, images, simulation parameters, etc, Fig.~\ref{rando}.
+\begin{figure}[t]
+\includegraphics[width=8cm]{emfRand1}
+\includegraphics[width=8cm]{emfRand3}
+\includegraphics[width=8cm]{collRand2}
+\includegraphics[width=8cm]{collRand3}
+\caption{Three randomized variations of the problems in Fig.~\ref{repre} and \ref{reprecoll}.
+The graphs in the emf problem are dynamically generated on-the-fly, the cars and the brickwall in the collision problem are randomly selected images.
+Each randomization leads to different answersfor different students. It should be noted that in this particular example, the collision problem
+became more difficult if the learner realized early on that in all combinations of Fig.~\ref{reprecoll}, the combined object after the collision will be at rest
+-- the corresponding constraints could have been implemented in the randomizing problem.\label{rando}}
+\end{figure}
+
+Over the years, the system added a learning content management system and standard course management features, such as communications, gradebook, etc., which are comparable to commercial course management systems, such as
+BlackBoard, WebCT, or ANGEL. SeeRefs.~\cite{features,edutools} for an overview of features, and comparisons to other systems.
+In addition to standard features, the LON-CAPA delivery and course management layer is designed around STEM education, for example:
+support for mathematical typesetting throughout (\LaTeX\ inside of XML) -- formulas are rendered on-the-fly,
+and can be algorithmically modified through the use of variables inside formulas; integrated GNUplot support, such that graphs can be rendered on-the-fly,
+and allowing additional layered labeling of graphs and images; support for multi-dimensional symbolic math answers; and full support of physical units.
+
+Relevant to this project is the finding that peer-communication soon emerged as an essential feature of the learning process, which led to an expansion of LON-CAPA's internal communication features. Of particular interest was the
+finding that different content representations led to significantly different peer-interactions. In a recent study of online discussions~\cite{discpaper} within our project, the majority of the discussion contributions were of
+type surface-level or procedural, followed by emotional contributions. The vast majority of discussion contributions had the feature of being solution-oriented, yet a considerable number dealt with the physics of the problems.
+\begin{figure} [t]
+\begin{center}
+\includegraphics[width=80mm]{KortemeyerFig5}
+\includegraphics[width=80mm]{KortemeyerFig6}
+\end{center}
+\caption{\label{fig:gradecorrel}Prominance of discussion contribution characteristics by student grade (left panel) and question difficulty (right panel).}
+\end{figure}
+
+{\bf Student Course Grade:} Significant differences as a function of course grade appeared when considering the classes of discussions (subsection~\ref{subsec:disccat}).
+Figure~\ref{fig:gradecorrel} shows the outcome of this study by discussion superclass. As an example, the figure is to be interpreted this way: within the indicated errors,
+55 percent of a 3.0 student's discussion contributions were solution-oriented.
+The lines represent second-order polynomial fits to the data. The results are not surprising, but verify the validity of the classification approach, which will also be used for this project.
+At the same time, the results confirm that conceptual and physics-related discussions are positively correlated with success in the course, while solution-oriented discussion contributions are strongly negatively correlated.
+While cause and effect may be arguable, in this project, particular attention needs to be paid to question properties that elicit either the desirable or undesirable discussion patterns.
+
+{\bf Influence of Question Types:} Different question types were found to result in different associated discussion patterns. Discussions on a procedural level are more prominent for numerical problems than for any other
+problem type. Solution-oriented discussions are more prominent for multiple-choice style questions, frequently in an effort to short-circuit the conceptual reasoning: it was found that students in this
+simple question type use the discussion space to reverse-engineer the randomization process by copying-and-pasting the their correct solutions.
+The prominance of conceptual discussions is significantly lower in single-response multiple-choice (the type currently used in peer-instruction)
+and numerical problems than in the other problem types. In the earlier study by Kashy~\cite{kashyd01}, it was also found that mastery of these same question types does not predict overall performance on the final exam
+as well as other question types. Multiple-choice problems that do not involve numbers are frequently simply called ``conceptual'' questions, but in~\cite{discpaper}, it was found that they do not {\it necessarily} lead to
+conceptual discussions. Ranking questions showed very favorable discussion patterns, but their sample size in~\cite{discpaper} has been too small to make definitive statements.
+Differences between ``conventional'' and representation-translation problems were small, while Kashy~\cite{kashyd01} found significant differences,
+and McDermott~\cite{mcdermott} and Beichner~\cite{beichner} pointed out the significance of the translation step. It should be noted that the earlier study~\cite{kashyd01} dealt with a relatively small set of
+representation-translation problems, some of which involved non-static time-evolving simulations as data-source, while in~\cite{discpaper}, none of the simulation-based problemswere assigned. Within this project, we aim to deploy
+Physlets~\cite{physlets} in the classroom, and expect statistically more significant data regarding their impact.
+
+{\bf Influence of Question Difficulty:} The right panel of Fig.~\ref{fig:gradecorrel} shows the prominance of different discussion contributions as a function of questiondifficulty.
+Very easy problems can elicit a high level conceptual discussions, and so can problems of mid-range difficulty. As problems become more difficult, there is no significant gain in conceptual discussions.
+
+An important feature of LON-CAPA for this study is its scalability and ability to absorb peak-workloads through cluster-wide load balancing. On campus of MSU, LON-CAPA has been used for timed online ``take-home'' exams by 400 students submitting 50 questions each within one hour.
+
The project developed into a content sharing network of more than 15 institutions of higher education including community colleges and four-year institutions, as well as over 15 middle and high schools. In addition, LON-CAPA houses commercial textbook content from seven major publishing companies, and a commercial service company was established around the product at the end of 2004. The project maintains a gateway server to the National Science Digital Library, and the LON-CAPA shared resource pool is searchable and accessible from {\tt http://nsdl.org/}.
LON-CAPA is used by approximately 25,000 students every semester.
Gerd Kortemeyer is Co-PIs on the current NSF-CCLI-ASA grant Diagnostic Question Clusters: Development and Testing in Introductory Geology and Biology (\#0243126, \$491,606, September 15, 2003 through August 31, 2006) to develop diagnostic questions for college students in both biology and geology. In the ASA project, a pool of peer-reviewed, diagnostic question clusters to assess students' understanding will be developed, including tools for analysis, peer review, and online publication of these question clusters.
-
+\subsection{Harvard Group}
The group of Eric Mazur has been supported by the National Science Foundation to develop, evaluate, and produce
materials for effective undergraduate science pedagogy and assessment since 1993
(DUE \#925407, Peer Instruction: Stimulating renewed interest in physics and other
science and engineering courses,
\$223,500,
2/1/93 - 1/31/96).
+
+We have applied peer-instruction in both the calculus-based and the algebra-based introductory physics courses for non-majors at Harvard University.
+Instructors have adopted the method across a variety of disciplines and courses, including senior-level courses, at a large number of institutions nationwide. Substantial gains in
+student achievement when comparing courses taught using peer-instruction to those taught with
+traditional pedagogy have been documented. These gains have been determined by a number of measures,including student
+mastery of content \cite{mref1,mref2,mref3,mref4,mref5,mref6,mref7,mref8,mref9,mref10}.
+The trend in improving student understanding proves to be particularly beneficial to female students, whose performance increases substantially, when taught using this
+interactive method \cite{mref13}.
+
+\begin{figure}[t]\begin{center}
+\includegraphics[width=2in]{fcipre}
+\includegraphics[width=2in]{fcipost}
+\end{center}\caption{Pre- and post-scores on the Force Concept Inventory of three courses at Harvard.\label{prepostfci}}
+\end{figure}
+Our ten years of experience with peer-instruction, as well as feedback from about 400 other instructors whohave used peer-instruction~\cite{mref27}, indicate that it is a successful way to actively engage students in large classes.
+Moreover, actively engaging students during class with a method such as peer-instruction leads to significant gains in conceptual understanding, as measured with standard conceptual instruments. Students in our calculus-based
+introductory physics course achieve Force Concept Inventory gains that are roughly twice those ofstudents in the same course taught traditionally,
+a level of improvement typical of a variety of interactive engagement strategies in physics~\cite{mref28}. Students also show comparable or improved quantitative problem-solving
+skills, despite a reduced emphasis on problems in class~\cite{mref11,mref12}.
+Research on collaborative education nearly universally indicates that collaborative work is more
+effective than passive learning. Our experiences with peer-instruction, as well as those of many others, who have
+responded to our survey, show peer-instruction to be an effective collaborative approach to learning.
+
+The primary resource needed for teaching with peer-instruction is a supply of suitable ConcepTests (CTs) -questions that test students' understanding of the basic concepts covered \cite{mref11}. We have developed and
+refined over 1,000 CTs for use in introductory physics courses. These CTs are freely available to instructors through the ILT web site (detailed below), together with over 400 additional CTs that have been contributed by others.
+An indicator of the rapid spread of the method is the availability of books with ConcepTests for chemistry, astronomy, and calculus courses.
The group developed Project Galileo (DUE \#9554870, On-line server of educational resources, \$800,000, 3/1/96-3/1/00; DUE \#9980802 Creating a community of Peer Instruction users: dissemination and
-electronic resources, \$290,000, 4/01/00-3/31/02), a store of extensive resources for
-interactive learning pedagogies, targeting both large and small classroom teaching techniques, which areavailable to the entire teaching community.
+electronic resources, \$290,000, 4/01/00-3/31/02), as a store of extensive resources for these
+interactive learning pedagogies, targeting both large and small classroom teaching techniques, which are available to the entire teaching community.
+
Using funds from a NSF Director's Distinguished Teaching Scholar Award (DUE \#123899, Distinguished Teaching Scholar Award: On-line Resources for Teaching With Peer Instruction, \$305,000, 9/15/01-8/31/05),
-we created the Interactive Learning Toolkit, a learning management system that allows instructors to implement several proven innovative teaching techniques and to share and review materials they create for these techniques.
+we created the Interactive Learning Toolkit (ILT), a learning management system that allows instructors to implement several proven innovative teaching techniques and to share and review materials they create for these techniques.
+The ILT features a set of tools that allow an instructor to structure and create content for their class and then analyze the feedback of the students.
+In order to 'free up' precious class time, the ILT offers an pre-class reading assignment tool.
+To help improve the interaction between students and instructor,a face book has been developed in ILT. What this means is that anywhere a student's name appears in the ILT, it links to their
+picture and also to a portal page showing their progress in all aspects of the course. This novel tool helps the instructor to become familiar with each student, helping improve individual interaction and to quickly
+identify students who might be struggling in the course. This is of particular pedagogical value in large classes, where students normally remain anonymous.
+Most importantly, the ILT provides a location to warehouse course content, like ConcepTests and Reading assignments, so it can be shared by the entire community of instructors.
+A simple rights management system allows the instructor to either maintain their copyright or place their material in the public domain.
+
The ILT is currently in use at a number of institutions nationwide, including Vanderbilt, University of Southern California, University of Massachusetts-Boston, Salem State College, Massachusetts Institute of Technology,
-Swarthmore College, with a student user base of several thousand students per semester.
+Swarthmore College, with a student user base of several thousand students per semester. The Harvard group recently started a collaboration with the group of Bill Junkin at Erskine College (also collaborators on this project),
+which develops the Beyond Question (BQ) system.
+BQ collects student responses electronically from various types of consumer devices~\cite{bq1,bq2}. BQ polls
+student responses and provides these data immediately for the instructor as needed.
+The software not only reads the output from electronic response units, but also can be used simultaneously -- with cellphones,
+PDAs and laptops, which can provide more comprehensive data than single-digit multiple-choice responses provided by current classroom communication systems. The strength of BQ is this flexibility,
+which makes it an 'easy to use' package for almostany teaching environment.
\section{Methodology}\label{method}
@@ -245,53 +350,6 @@
\end{tabular}
\end{table}
-\subsubsection{Previous Results of Discussion Analysis}\label{prevdiscrev}
-The MSU group previously carried out discussion analyses of learners collaborating around introductory Computer Science problems, as well as asynchronously around online homework problems in introductory Physics.
-While it is to be researched if any results transfer to in-class discussion settings, we are encouraged by statistically significant outcomes of these earlier studies.
-In the study of online discussions~\cite{discpaper}, the majority of the discussion contributions were of type surface-level or procedural, followed by emotional
-contributions. The vast majority of discussion contributions had the feature of being solution-oriented,
-yet a considerable number dealt with the physics
-of the problems.
-
-
-\begin{figure} [t]
-\begin{center}
-\includegraphics[width=80mm]{KortemeyerFig5}% Here is how to import EPS art
-\includegraphics[width=80mm]{KortemeyerFig6}
-\end{center}
-\caption{\label{fig:gradecorrel}Prominance of discussion superclasses by student grade (left panel) and question difficulty (right panel).}
-\end{figure}
-
-{\bf Student Course Grade:}
-Significant differences as a function of course grade appeared when considering the classes of discussions (subsection~\ref{subsec:disccat}).
-Figure~\ref{fig:gradecorrel}
-shows the outcome of this study by discussion superclass. As an example, the figure is to be interpreted this way: within the indicated errors,
-55 percent of a 3.0 student's discussion contributions were solution-oriented. The lines represent second-order polynomial fits to the data.
-
-The results are not surprising, but verify the validity of the classification approach. At the same time, the results confirm that conceptual and
-physics-related discussions are positively correlated with success in the course, while solution-oriented discussion contributions are strongly negatively correlated.
-While cause and effect may be arguable, in this project,
-particular attention needs to be paid to question properties that elicit either the desirable or undesirable discussion patterns.
-
-{\bf Influence of Question Types:}
-Different question types were found to result in different associated discussion patterns. Discussions on a procedural level are more prominent for numerical problems than for any other problem type. Solution-oriented discussions are more prominent for multiple-choice style questions, frequently in an effort to short-circuit the conceptual reasoning: it was found that students in this simple question type use the discussion space to reverse-engineer the randomization process by copying-and-pasting the their correct solutions (see the surface-level solution-oriented example in Table~\ref{table:examples}).
-
-The prominance of conceptual discussions is significantly lower in single-response multiple-choice (the type currently used in peer-instruction)
-and numerical problems than in the other problem types. In the
-earlier study by Kashy~\cite{kashyd01}, it was also found that mastery of these same question types does not predict overall performance on the final exam as well as other question types.
-Multiple-choice problems that do not involve numbers are frequently simply called ``conceptual'' questions, but in~\cite{discpaper},
-it was found that they do not {\it necessarily} lead to conceptual discussions.
-Ranking questions showed very favorable discussion patterns, but their sample size in~\cite{discpaper} has been too small to make definitive statements.
-individual discussion behavior and performance in the course clearly exists.
-
-Differences between ``conventional'' and representation-translation problems were small, while Kashy~\cite{kashyd01} found significant differences, and McDermott~\cite{mcdermott} and Beichner~\cite{beichner} pointed out the significance of the translation step. It should be noted that the earlier study~\cite{kashyd01} dealt with a relatively small set of
-representation-translation problems, some of which involved non-static time-evolving simulations as data-source, while in~\cite{discpaper}, none of the simulation-based problems
-were assigned. Within this project, we aim to deploy Physlets~\cite{physlets} in the classroom, and expect statistically more significant data regarding their impact.
-
-{\bf Influence of Question Difficulty:} The right panel of Fig.~\ref{fig:gradecorrel} shows the prominance of different discussion contributions as a function of question
-difficulty. Very easy problems can elicit a high level conceptual discussions,
-and so can problems of mid-range difficulty. As problems become more difficult, there is no significant gain in conceptual discussions.
-
\subsubsection{Procedure}
Student helpers will be trained and assigned to student groups during lecture to document the discussions using the coding scheme described in subsection~\ref{subsec:disccat}. They will be provided with worksheets to quickly tabulate contributions. Results will be stored in conjunction with the statistical data gathered from each question. Analysis will be carried out as described in \ref{prevdiscrev}.
@@ -329,111 +387,10 @@
Since in addition, questions are randomizing, we are able to include some of the same questions used in class on exams and
quizzes. A similar study was previously conducted by Kashy~\cite{kashyd01} for homework questions.
-\subsubsection{Previous Results}
-We have applied
-peer-instruction in both the calculus-based and the algebra-based introductory physics courses for non-majors
-at Harvard University. Instructors nationwide have adopted the method across a variety of disciplines and
-courses, including senior-level courses, at a large number of institutions nationwide. Substantial gains in
-student achievement when comparing courses taught using peer-instruction to those taught with
-traditional pedagogy
-have been documented. These gains have been determined by a number of measures,
-including student
-mastery of content \cite{mref1,mref2,mref3,mref4,mref5,mref6,mref7,mref8,mref9,mref10}.
-The trend in improving student understanding proves to be particularly
-beneficial to female students, whose performance increases substantially, when taught using this
-interactive method \cite{mref13}.
-
-\begin{figure}[t]
-\begin{center}
-\includegraphics[width=2in]{fcipre}
-\includegraphics[width=2in]{fcipost}
-\end{center}
-\caption{Pre- and post-scores on the Force Concept Inventory of three courses at Harvard.\label{prepostfci}}
-\end{figure}
-
-Our ten years of experience with peer-instruction, as well as feedback from about 400 other instructors who
-have used peer-instruction~\cite{mref27}, indicate that it is a successful way to actively engage students in large classes.
-Moreover, actively engaging students during class with a method such as peer-instruction leads to significant gains in
-conceptual understanding, as measured with standard conceptual instruments. Students in our calculusbased
-introductory physics course achieve Force Concept Inventory gains that are roughly twice those of
-students in the same course taught traditionally, a level of improvement typical of a variety of interactive
-engagement strategies in physics~\cite{mref28}. Students also show comparable or improved quantitative problemsolving
-skills, despite a reduced emphasis on problems in class~\cite{mref11,mref12}.
-Research on collaborative education nearly universally indicates that collaborative work is more
-effective than passive learning. Our experiences with peer-instruction, as well as those of many others, who have
-responded to our survey, show peer-instruction to be an effective collaborative approach to learning.
-
-
-\section{Existing Infrastructure}
-\subsection{Existing Material}
-The primary resource needed for teaching with peer-instruction is a supply of suitable ConcepTests (CTs) -
-questions that test students' understanding of the basic concepts covered \cite{mref11}. We have developed and
-refined over 1,000 CTs for use in introductory physics courses. These CTs are freely available to
-instructors through the ILT web site (detailed below), together with over 400 additional CTs that have
-been contributed by others. An indicator of the rapid spread of the method is the availability of books
-with ConcepTests for chemistry, astronomy, and calculus courses. We are currently in the process of
-adding this material to the ILT web site.
-
+\section{Implementation of the Resource Pool}
In LON-CAPA, the underlying distributed multimedia content repository spans across all of the currently over 30 participating institutions, and currently contains over 60,000 learning content resources, including more than 18,000 personalized homework problems. Disciplines include astronomy, biology, business, chemistry, civil engineering, computer science, family and child ecology, geology, human food and nutrition, human medicine, mathematics, medical technology, physics, and psychology. Any content material contributed to the pool is immediately available and ready-to-use within the system at all participating sites, thus facilitating dissemination of curricular development efforts. A large fraction of these resources are also available through the gateway to the National Science Digital Library (NSDL).
-\subsection{Existing System Functionality}\label{existing}
-Between the collaborators, a number of technological components exist that can be used to build the foundation for the needed functionality.
-
-\subsubsection{Interactive Learning Toolkit (ILT)}
-The Interactive Learning Toolkit is an open-source learning management system that supports research-based interactive
-learning pedagogies, like Just In Time Teaching and Peer Instruction. The ILT development group is located at Harvard
-University.
-
-The ILT features a set of tools that allow an instructor to structure and create content for their class and then analyze
-the feedback of the students. In order to 'free up' precious class time, the ILT offers an pre-class reading assignment tool.
-
-To help improve the interaction between students and instructor,
-a face book has been developed in ILT. What this means is that anywhere a student's name appears in the ILT, it links to their
-picture and also to a portal page showing their progress in all aspects of the course. This novel tool helps the instructor to
-become familiar with each student, helping improve individual interaction and to quickly identify students who might be struggling
-in the course. This is of particular pedagogical value in large classes, where students normally remain anonymous.
-
-Most importantly, the ILT provides a location to warehouse course content, like ConcepTests and Reading assignments, so it can be
-shared by the entire community of instructors. A simple rights management system allows the instructor to either maintain their
-copyright or place their material in the public domain.
-
-\subsubsection{Beyond Question (BQ)}
-The BQ system collects student responses electronically from various types of consumer devices~\cite{bq1,bq2}. BQ polls
-student responses and provides these data immediately for the instructor as needed.
-The software not only reads the output from electronic response units, but also can be used simultaneously - with cellphones,
-PDAs and laptops, which can provide more comprehensive data than single-digit multiple-choice responses provided by current
-classroom communication systems. The strength of BQ is this flexibility, which makes it an 'easy to use' package for almost
-any teaching environment. The BQ development group is located at Erskine College.
-
-\subsubsection{The Learning{\it Online} Network with CAPA (LON-CAPA)}\label{loncapa}
-The Learning{\it Online} Network with Computer-Assisted Personalized Approach ({\tt http://www.lon-capa.org/}) is a distributed learning content management, course management, and assessment system, and also the model system of the current NSF-ITR grant, see Sect.~\ref{results}.
-
-LON-CAPA is open-source (GNU General Public License) freeware, there are no licensing costs associated. Both aspects are important for the success of this project: the open-source nature of the system allows researchers to modify and adapt the system in order to address research needs, and the freeware character allows easier dissemination of results, in particular adaptation and implementation at other universities.
-
-LON-CAPA is designed around the concept of easy sharing and re-use of learning resources, as well as advanced assessment capabilities in science courses.
-The system started in 1992 as a system to give personalized homework to students in introductory physics courses. ``Personalized" means that each student sees a different version of the same computer-generated problem: different numbers, choices, graphs, images, simulation parameters, etc,
-Fig.~\ref{rando}.
-
-\begin{figure}[t]
-\includegraphics[width=8cm]{emfRand1}
-\includegraphics[width=8cm]{emfRand3}
-\includegraphics[width=8cm]{collRand2}
-\includegraphics[width=8cm]{collRand3}
-
-\caption{Three randomized variations of the problems in Fig.~\ref{repre} and \ref{reprecoll}. The graphs in the emf problem are dynamically generated on-the-fly, the cars and the brick
-wall in the collision problem are randomly selected images. Each randomization leads to different answers
-for different students. It should be noted that in this particular example, the collision problem became more difficult if the learner realized early on that in all
-combinations of Fig.~\ref{reprecoll}, the combined object after the collision will be at rest -- the corresponding constraints could have been implemented in the randomizing problem.
-\label{rando}}
-\end{figure}
-
-Over the years, the system added a learning content management system and standard course management features, such as communications, gradebook, etc., which are comparable to commercial course management systems, such as BlackBoard, WebCT, or ANGEL. See
-Refs.~\cite{features,edutools} for an overview of features, and comparisons to other systems.
-
-In addition to standard features, the LON-CAPA delivery and course management layer is designed around STEM education, for example: support for mathematical typesetting throughout (\LaTeX\ inside of XML) -- formulas are rendered on-the-fly, and can be algorithmically modified through the use of variables inside formulas; integrated GNUplot support, such that graphs can be rendered on-the-fly, and allowing additional layered labeling of graphs and images; support for multi-dimensional symbolic math answers; and full support of physical units.
-An important feature of LON-CAPA for this study is its scalability and ability to absorb peak-workloads through cluster-wide load balancing. On campus of MSU, LON-CAPA has been used for timed online ``take-home'' exams by 400 students submitting 50 questions each within one hour.
-\subsection{Resource-Pool Organization}
The ConcepTest library will be ported to the LON-CAPA system and stored in the shared resource pool of the LearningOnline Network with CAPA (subsection~\ref{loncapa}), which
offers scalable cross-institutional content and rights management features. Where appropriate, gateways will be established to have different system components access the same
original content.
--www1116429268--