[LON-CAPA-cvs] cvs: modules /gerd/roleclicker description.tex
www
lon-capa-cvs@mail.lon-capa.org
Mon, 09 May 2005 16:56:57 -0000
This is a MIME encoded message
--www1115657817
Content-Type: text/plain
www Mon May 9 12:56:57 2005 EDT
Modified files:
/modules/gerd/roleclicker description.tex
Log:
Condense
--www1115657817
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20050509125657.txt"
Index: modules/gerd/roleclicker/description.tex
diff -u modules/gerd/roleclicker/description.tex:1.8 modules/gerd/roleclicker/description.tex:1.9
--- modules/gerd/roleclicker/description.tex:1.8 Mon May 9 09:47:19 2005
+++ modules/gerd/roleclicker/description.tex Mon May 9 12:56:56 2005
@@ -114,7 +114,21 @@
\section{Methodology}\label{method}
-The effectiveness of the extensions to peer-instruction will be evaluated by both with focus on process and on learning outcomes.
+\subsection{Establishment of Initial Conditions}
+Many educational studies result in ``no-significant-difference"~\cite{russell}, and particularly the study of question type effectiveness (Sect.~\ref{effect}) may well yield the same result. Many variables may influence the impact of a particular sample of representative problems of a particular type for a particular learner, and it is imperative to understand as much of the ``initial conditions" as possible, since the validity of the hypothesis may depend on them.
+
+\subsubsection{Learner Attitudes, Beliefs, and Expectations}
+Instruments have been developed to assess epistemological beliefs, for example the Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument~\cite{EBAPS}. Related to epistemological beliefs are learnerÕs expectations and attitudes, and of particular interest is the Maryland Physics Expectations (MPEX) survey~\cite{MPEX}.
+
+\subsubsection{Learner Knowledge about the Topic}\label{prepost}
+
+
+The effectiveness of the extensions to peer-instruction will be evaluated both with focus on process and on learning outcomes.
+
+
+
+
+
\subsection{Process-Oriented Evaluation}
The process-oriented evaluation will focus on the actual discussion process. Since currently no baseline data exists for this study, we will assess the quality of student discussion both
before and after the introduction of extensions to the current peer-instruction technique.
@@ -127,7 +141,7 @@
with multiple-choice non-numerical problems having the lowest correlation, and numerical/mathematical problems that require a translation of representation having the highest.
Steinberg~\cite{steinberg} also analyzed student performance on multiple-choice diagnostics and open-ended exam problems, and found that while those correlate in general, for certain students
and certain questions, responses differ greatly.
-For this project, we chose a finer-grained classification scheme of question types: Redish~\cite{redish} identifies eight classes and features of exam and homework questions,
+For this project, we are choosing a finer-grained classification scheme of question types: Redish~\cite{redish} identifies eight classes and features of exam and homework questions,
and an adapted version of this scheme will be used:
\begin{description}
\item[Multiple-choice and short-answer questions] The most basic and most easily computer-evaluated type of question, representing the conventional (typical back-of-chapter textbook) problem.
@@ -160,45 +174,9 @@
\item[Qualitative questions] This type of questions asks students to make judgments about physical scenarios, and in that respect are somewhat similar to ranking questions. While the questions themselves are of the type ``Is this high enough?" or ``Can we safely ignore \ldots?," they often do require at least ``back-of-the-envelope" calculations to to give informed answers. As in the case of estimation problems, students do have to explain their reasoning, but the question itself is usually more structured, and at least the initial answer is more easily evaluated by a computer.
\item[Essay questions] These are ``explain why" questions. A certain scenario is presented, and students are asked to explain why it turns out the way it does. Students are not asked to recall a certain law --- it is given to them. Instead, they are asked to discuss its validity.
\end{description}
-The three courses did not include estimation, qualitative, and essay problems, which cannot be graded automatically within the online system.
-Table~\ref{table:problemcat} shows the classification distribution of the online
-problems available for this project.
-
-\begin{table*}
-\caption{Classification of the online questions according the classification scheme described in
-subsection~\ref{subsec:problemcat} (adapted from Redish~\cite{redish}). The columns denote the
-different question types, while the rows denote the features of required representation translation and
-context-based reasoning.\label{table:problemcat}}
-\begin{ruledtabular}
-\begin{tabular}{lccccccc|l}
-&\multicolumn{4}{c}{Multiple-choice and short-answer}&Mult.-choice mult.-resp.&Ranking&Click-on-image&\\
-&Multiple-choice&Tex\-tual&Nume\-rical&For\-mula&&&\\
-\hline
-``Conventional''& 10& & 355& 3& 54& 4& 2&428\\
-Rep-Trans & 7& & 38& & 16& 1& 7&69\\
-Context-based & & & & & & & &0\\\hline
- & 17& & 393& 3& 70& 5& 9&497
-\end{tabular}
-\end{ruledtabular}
-\end{table*}
-Of the 497 online questions available for this study, none required context-based reasoning, and none expected
-a free-form short textual answer. Approximately 14 percent of the questions required representation translation.
-The vast majority of questions were conventional numerical problems, which expect
-a numerical answer with associated physical unit.
-
-In addition, for every question, its
-difficulty index was computed according to the formula
-\begin{equation}\label{eqn:diffidx}
-\mbox{Difficulty Index}=10\left(1-\frac{N_{\mbox{correct}}}{N_{\mbox{attempts}}}\right)
-\end{equation}
-where $N_{\mbox{correct}}$ is the total number of correct solution in the course, and $N_{\mbox{attempts}}$ is the total number of
-correct and incorrect solution submissions (the system allows multiple attempts to arrive at the correct solution, see
-subsection~\ref{subsec:system}). If all submissions were correct, meaning, every student would have solved the problem
-correctly on the first attempt, the difficulty index would be 0. If none of the submissions were correct, the index would be 10.
-
-
+The same classification scheme was successfully used for an analysis of online asynchronous discussions~\cite{discpaper}.
\subsubsection{\label{subsec:disccat}Discussion Classification}
-Student discussion entries were classified into three types and four features. The four types are
+Student discussion entries are classified into three types and four features. The four types are
\begin{description}
\item[Emotional] - discussion contributions were classified as ``emotional" if they mostly communicated opinions,
complaints, gratitude, feelings, etc. Two subtypes were ``positive" and ``negative."
@@ -217,9 +195,9 @@
\item[Mathematical] - the contribution deals mostly with the mathematical aspects of the problem.
\item[Physics] - the contribution deals mostly with the physics aspects of the problem.
\end{description}
-Table~\ref{table:examples} shows examples of contributions and their classification.
-\begin{table*}
-\caption{Examples of discussion contribution types and features.\label{table:examples}}
+Table~\ref{table:examples} shows examples of contributions from the study on online discussions~\cite{discpaper} and their classification.
+\begin{table}
+\caption{Examples of discussion contribution types and features~\cite{discpaper}.\label{table:examples}}
\begin{tabular}{l|p{3.9cm}|p{3.9cm}|p{3.9cm}|p{3.9cm}}
&Unrelated&Solution&Math&Physics\\\hline
@@ -259,32 +237,8 @@
&
I have the correct answer, but I don't understand why it is correct. Why would there be an acceleration at the ball's highest point? Why wouldn't it be zero?
\end{tabular}
-\end{table*}
-Discussion contributions were always classified as a whole, and since they were fairly short, they mostly fell clearly into one of the classes. If a longer contribution had aspects of more than one class, it was characterized by
-the class that its majority fell into. Discussion contributions by teaching assistants and instructors were not
-considered. Also, the correctness of the posting was not considered, e.g., a discussion entry was considered ``conceptual'' even if it drew the wrong conclusions.
- Table~\ref{table:disccat} shows the distribution of the available discussion contributions.
-\begin{table}
-\caption{Classification of the online discussion contributions according the classification scheme described in
-subsection~\ref{subsec:disccat}. The columns denote the different discussion types and subtypes, while the
-rows denote the
-features.\label{table:disccat}}
-\begin{tabular}{lcccccccc|l}
-&\multicolumn{2}{c}{Emotional}
-&\multicolumn{2}{c}{Surface}
-&\multicolumn{2}{c}{Procedural}
-&\multicolumn{2}{c}{Conceptual}&\\
-
- &Pos&Neg&Q&A&Q&A&Q&A&\\
- Unrelated& 71& 54& 10& 1& & & 1& &137\\
- Solution & 279& 185& 601& 341& 353& 456& 12& 3&2230\\
- Math & 1& 6& 49& 36& 73& 87& 3& 6&261\\
- Physics & & 14& 85& 81& 170& 190& 100& 126&766\\\hline
- & 351& 259& 745& 459& 596& 733& 116& 135&3394
-\end{tabular}
\end{table}
-
In addition, the following
superclasses are considered:
\begin{description}
@@ -294,35 +248,18 @@
\item[Type and feature sums] - number of all related contributions belonging to a certain type, subtype, or feature.
\end{description}
-The majority of the discussion contributions were of type surface-level or procedural, followed by emotional
+In the study of online discussions~\cite{discpaper}, the majority of the discussion contributions were of type surface-level or procedural, followed by emotional
contributions.
The vast majority of discussion contributions had the feature of being solution-oriented,
yet a considerable number dealt with the physics
of the problems.
-
-
-\section{Results of Analysis by Student}
-\subsection{Participation}
-\begin{figure*}
-\includegraphics[width=160mm]{KortemeyerFig4}% Here is how to import EPS art
-\caption{\label{fig:contribBinned}Number of students versus number of discussion contributions.}
-\end{figure*}
-Within the first semester calculus-based course, an analysis by student was performed. Out of the 211 students in the course,
-138 students (65 percent) contributed at least one discussion posting over the course of the semester. Figure~\ref{fig:contribBinned} shows the distribution
-of number of discussion contributions over the course of the semester. Most students who participated made between one and ten contributions, but one student made
-66 postings.
-It is not possible to find out which percentage students {\it read} the discussions, since discussion are automatically attached to the questions and always visible.
-The average number of postings per student was $5\pm0.7$. Women had a significantly higher average number of postings than men:
-each female student contributed an average of $5.9\pm1$ postings, while each male student contributed an average of $3.7\pm0.7$ postings.
-\subsection{Grade-Dependence of Discussion Contributions\label{subsec:gradedep}}
-The average grade in the course was $3.21\pm0.05$, with men and women achieving equally high grades (men: $3.29\pm0.08$; women: $3.17\pm0.05$).
-No correlation could be found between the average number of discussion postings and the grade in the course --- in terms of absolute
-numbers, within statistical errors, students with high and low grades in the course participated equally in the discussions. The positive correlation between participation in the
-this ``moderated'' discussion forum and course grades~\cite{kashy03} could not be confirmed in this study.
+\subsubsection{Previous Results of Discussion Analysis}
\begin{figure}
\includegraphics[width=86mm]{KortemeyerFig5}% Here is how to import EPS art
\caption{\label{fig:gradecorrel}Prominance of discussion superclasses by grade.}
\end{figure}
+\begin{description}
+\item[Student Course Grade] -
Significant differences as a function of course grade appear when considering the classes of discussions (subsection~\ref{subsec:disccat}).
In this analysis, the percentage prominance of certain types and
features in students' cummulative contributions over the semester was analyzed. The individual percentage (relative) prominances were then averaged by grade.
@@ -340,59 +277,15 @@
At the same time, the results confirm that conceptual and physics-related discussions are positively correlated with success in the course, while solution-oriented discussion contributions are strongly negatively correlated. While cause and effect may be arguable, in the following
section~\ref{sec:question}, particular attention needs to be paid to question properties that elicit either the desirable or undesirable discussion behavioral patterns.
-
-\section{Results of Analysis by Question\label{sec:question}}
-\subsection{Influence of Question Difficulty}
-Each discussion contribution associated with a question was classified according to the types and features described in
-subsection~\ref{subsec:disccat}. As a measure of the prominence of a class in a given discussion,
-the number of contributions belonging to it is divided by the total number of contributions. The discussion characteristics of the problems were binned by their
-difficulty index and the average percentage plotted in figure~\ref{fig:diff}. Only superclasses are
-shown (subsection~\ref{subsec:problemcat}), namely the emotional climate (crosses), as well as all (questions and answers) related
-procedural
-(triangles) and conceptual (diamonds) contributions. As an example, the plot is to be interpreted in the following way: within the given
-error boundaries, for a question with difficulty index of six, ten percent of the online discussion is conceptual.
-\begin{figure}
-\includegraphics[width=92mm]{KortemeyerFig6}% Here is how to import EPS art
-\caption{\label{fig:diff}Discussion characteristics as a function of problem difficulty.
-}
-\end{figure}
-In addition, the data was fit using second order (procedural, long dashes) and third order (emotional climate, short dashes; conceptual, solid) polynomials.
-
-The greatest variation is found in the emotional climate of the discussion. As is to be expected, the climate is mostly positive
-for ``easy" questions, but then remains positive for a fairly wide range of problem difficulties until it becomes negative
-at a difficulty index of 7. Only six questions had a difficulty index of 9, and --- surprisingly --- none of these had
-associated emotional comments.
-
-For difficulty indizes beyond 3, the prominence of conceptual discussions increases. Surprisingly, it also increases for easier
-questions. This may be attributed to students feeling more confident discussing easier problems on a conceptual level, or simply
-in there being less of a need of procedural discussions.
-Overall, the prominence of conceptual discussions is disappointingly low, as it varies between 5 and 16 percent.
-
-Beyond a difficulty index of 5, within error boundaries, the prominence of conceptual discussions would be consistent with a constant 10 percent. If fostering them is a goal,
-and the emotional climate an indicator of ``pain,'' then beyond a difficulty index of 5 a significant increase in ``pain'' results in a non-significant gain.
-
-Across all difficulties, procedural contributions dominate the discussions, with relatively little significant variance around
-the 40 percent mark. The maximum occurs for questions with a difficulty index of 5.
-
-In figure~\ref{fig:diffnochat} the same analysis was carried out, but this time excluding all ``chat" contributions
-(subsection~\ref{subsec:problemcat}), i.e., only related non-emotional contributions were considered. The relative prominence of procedural and conceptual discussions systematically
-increases, but all observations from the full analysis remain valid. ``Chat'' mostly provides a constant background across all difficulty indices.
-\begin{figure}
-\includegraphics[width=92mm]{KortemeyerFig7}% Here is how to import EPS art
-\caption{\label{fig:diffnochat}Discussion characteristics as a function of problem difficulty, no considering ``chat."
-}
-\end{figure}
-
-\subsection{\label{subsec:qtype}Influence of Question Types}
+\item[Influence of Question Types] -
Each question was classified according to the types and features described in subsection~\ref{subsec:problemcat}, and each associated discussion entry according to~\ref{subsec:disccat}. As a measure of the prominence of a class in a given discussion,
the number of contributions belonging to it is divided by the total number of contributions.
Table~\ref{table:qtype} shows the percentage prominence of discussion contributions with a certain type or with certain features in the discussions associated with questions
that are of a certain type or have certain features.
-\begin{table*}
+\begin{table}
\caption{Influence of question types and features on discussions.
The values indicate the percentage prominence of the discussion superclasses, types, and features (columns) for discussions associated with questions of a certain
type or with certain features (rows). The values in brackets result from an analysis with ``chat'' excluded.\label{table:qtype}}
-\begin{ruledtabular}
\begin{tabular}{lcccccc}
&Emot. Clim.&Procedural&Solution&Math&Physics&Conceptual\\
Multiple Choice&-5$\pm$3&28$\pm$7 (29$\pm$8)&66$\pm$7 (74$\pm$7)&9$\pm$6 (9$\pm$6)&16$\pm$5 (17$\pm$5)&6$\pm$3 (7$\pm$3)\\
@@ -406,8 +299,7 @@
``Conventional''&4$\pm$1&42$\pm$1 (50$\pm$2)&55$\pm$1 (65$\pm$2)&7$\pm$1 (8$\pm$1)&23$\pm$1 (27$\pm$1)&9$\pm$1 (10$\pm$1)\\
Rep-Trans&-2$\pm$2&37$\pm$4 (45$\pm$4)&52$\pm$3 (63$\pm$4)&7$\pm$2 (9$\pm$2)&23$\pm$3 (28$\pm$3)&8$\pm$2 (10$\pm$2)\\
\end{tabular}
-\end{ruledtabular}
-\end{table*}
+\end{table}
Error boundaries on the emotional climate values are rather large and mostly include zero (neutral), indicating no significant preferences within the limited sample.
Yet, students clearly dislike multiple-choice questions, while they clearly like numerical answer problems. The data also indicates that students prefer ``conventional'' over
representation-translation problems.
@@ -435,7 +327,7 @@
It should be noted that the earlier study dealt with a relatively small set of
representation-translation problems, some of which involved non-static time-evolving simulations as data-source, while in this study, none of the simulation-based problems were assigned. A future study may need to consider the interpretation of time-evolving
simulations as a separate feature, once that more problems of this type exist in the resource pool.
-\subsection{Influence of course}
+\item[Influence of course]
Few significant differences could be found between the algebra-based and the calculus-based course:
\begin{itemize}
\item discussions in the algebra-based course had a significantly higher emotional
@@ -449,25 +341,25 @@
homework problems~\cite{lin}.
+\subsubsection{Interviews}
+We will interview focus groups of students regarding their experiences and perceived relative helpfulness of the different problem types, and ask them to also reflect on how they perceived these question types were influencing their problem-solving strategies. Pascarella~\cite{pascarella02} developed some frameworks for these interviews, which can be built upon.
+
+
\subsection{Outcome-Oriented Evaluation}
-
+\subsubsection{Pre-/Post-Discussion Answer Distribution}
+\subsubsection{Pre-/Post-Performance on Concept Inventories}
+We will use existing concept inventory surveys as both pre- and post-tests.
+The qualitative Force Concept Inventory~\cite{fci} and the quantitative companion Mechanical Baseline Test~\cite{hestenesmech} have been used in a large number of studies connected to the teaching of introductory mechanics. The Foundation Coalition has been developing a number of relevant concept inventories~\cite{foundation}, namely the Thermodynamics Concept Inventory, the Dynamics Concept Inventory, and the Electromagnetics Concept Inventory (with two subcomponents, namely Waves and Fields). Since these were designed from an engineering point of view, some adjustment might be necessary. In addition, the Conceptual Survey of Electricity and Magnetism (CSEM)~\cite{maloney} is available for the second semester course.
-%
-%
-% Copy-Paste
-%
-\section{Background}
-\subsection{Peer-Instruction}
-Since developing Peer-Instruction (PI), a collaborative and interactive teaching technique, in 1991, we
-have been 1) disseminating the technique, 2) gathering data on its effectiveness (see publication list
-below), and 3) developing web-based tools to help instructors implement the method. We have applied
+\subsubsection{Previous Results}
+We have applied
the method in both the calculus-based and the algebra-based introductory physics courses for non-majors
at Harvard University. Instructors nationwide have adopted the method across a variety of disciplines and
courses, including senior-level courses, at a large number of institutions nationwide. Substantial gains in
@@ -479,33 +371,7 @@
The trend in improving student understanding proves to be particularly
beneficial to female students, whose performance increases substantially, when taught using this
interactive method \cite{mref13}.
-The primary resource needed for teaching with PI is a supply of suitable ConcepTests (CTs) -
-questions that test students' understanding of the basic concepts covered \cite{mref11}. We have developed and
-refined over 1,000 CTs for use in introductory physics courses. These CTs are freely available to
-instructors through the ILT web site (detailed below), together with over 400 additional CTs that have
-been contributed by others. An indicator of the rapid spread of the method is the availability of books
-with ConcepTests for chemistry, astronomy, and calculus courses. We are currently in the process of
-adding this material to the ILT web site.
-Under NSF sponsorship, we developed Project Galileo4, a store of extensive resources for
-interactive learning pedagogies, targeting both large and small classroom teaching techniques, which are
-available to the entire teaching community. Using funds from a NSF Director's Distinguished Teaching
-Scholar Award, we created the Interactive Learning Toolkit, a learning management system that allows
-instructors to implement several proven innovative teaching techniques and to share and review materials
-they create for these techniques. The ILT is currently in use at a number of institutions nationwide,
-including Vanderbilt, University of Southern California, University of Massachusetts-Boston, Salem State
-College, Massachusetts Institute of Technology, Swarthmore College, with a student user base of several
-thousand students per semester.
-We also invested a great deal of effort disseminating our findings nationwide, as we feel that it is
-crucial to share the results of our research. In the last several years, Eric Mazur and other members of the
-group have given more than one hundred invited talks on PI in a variety of venues:
-\begin{itemize}
-\item Physics department colloquia at a wide range of institutions from large state universities to small
-liberal arts colleges and community colleges;
-\item Workshops for new faculty sponsored by the American Association of Physics Teachers and the NSFfunded
-Engineering Education Scholars program;
-\end{itemize}
-\subsection{Collaborative Learning}
There is a significant body of literature concerning theories of and best practices for collaborative
learning \cite{mref14}. In general, the motivation behind collaborative learning is to make students active
participants in the learning process, assigning them more responsibility for their own education. Students
@@ -566,8 +432,31 @@
Research on collaborative education nearly universally indicates that collaborative work is more
effective than passive learning. Our experiences with PI, as well as those of many others, who have
responded to our survey, show PI to be an effective collaborative approach to learning.
-
-\subsection{Interactive Learning Toolkit}
+
+
+\section{Materials Devolopment}
+The primary resource needed for teaching with PI is a supply of suitable ConcepTests (CTs) -
+questions that test students' understanding of the basic concepts covered \cite{mref11}. We have developed and
+refined over 1,000 CTs for use in introductory physics courses. These CTs are freely available to
+instructors through the ILT web site (detailed below), together with over 400 additional CTs that have
+been contributed by others. An indicator of the rapid spread of the method is the availability of books
+with ConcepTests for chemistry, astronomy, and calculus courses. We are currently in the process of
+adding this material to the ILT web site.
+
+Under NSF sponsorship, we developed Project Galileo4, a store of extensive resources for
+interactive learning pedagogies, targeting both large and small classroom teaching techniques, which are
+available to the entire teaching community. Using funds from a NSF Director's Distinguished Teaching
+Scholar Award, we created the Interactive Learning Toolkit, a learning management system that allows
+instructors to implement several proven innovative teaching techniques and to share and review materials
+they create for these techniques. The ILT is currently in use at a number of institutions nationwide,
+including Vanderbilt, University of Southern California, University of Massachusetts-Boston, Salem State
+College, Massachusetts Institute of Technology, Swarthmore College, with a student user base of several
+thousand students per semester.
+\section{Implementation}
+
+\subsection{Existing System Functionality}
+
+\subsubsection{Interactive Learning Toolkit}
Two years ago, we published the result of an online survey of PI users in diverse settings. \cite{mref27} This
survey alerted us to some key areas that instructors found challenging, including the time taken to
structure and organize teaching with PI efficiently, the lack of teaching materials (specifically CTs), use
@@ -626,13 +515,13 @@
-\subsection{The Learning{\it Online} Network with CAPA}\label{loncapa}
+\subsubsection{The Learning{\it Online} Network with CAPA}\label{loncapa}
The Learning{\it Online} Network with Computer-Assisted Personalized Approach ({\tt http://www.lon-capa.org/}) is a distributed learning content management, course management, and assessment system, and also the model system of the current NSF-ITR grant, see Sect.~\ref{results}.
LON-CAPA's core development group is located at MSU, and in addition to faculty members, has a staff of three fulltime programmers, two user support staff, one technician, one graduate student, and one project coordinator. The LON-CAPA group also offers training and support for adopters of the system.
LON-CAPA is open-source (GNU General Public License) freeware, there are no licensing costs associated. Both aspects are important for the success of this project: the open-source nature of the system allows researchers to modify and adapt the system in order to address research needs, and the freeware character allows easier dissemination of results, in particular adaptation and implementation at other universities.
-\subsubsection{Shared Distributed Content Repository}
+\subsubsubsection{Shared Distributed Content Repository}
LON-CAPA is designed around the concept of easy sharing and re-use of learning resources.
In LON-CAPA, the underlying distributed multimedia content repository spans across all of the currently over 30 participating institutions, and currently contains over 60,000 learning content resources, including more than 18,000 personalized homework problems. Disciplines include astronomy, biology, business, chemistry, civil engineering, computer science, family and child ecology, geology, human food and nutrition, human medicine, mathematics, medical technology, physics, and psychology. Any content material contributed to the pool is immediately available and ready-to-use within the system at all participating sites, thus facilitating dissemination of curricular development efforts (Sect.~\ref{matdev}). A large fraction of these resources are also available through the gateway to the National Science Digital Library (NSDL).
@@ -646,7 +535,7 @@
LON-CAPA provides highly customizable access control for such resources, and has a built-in key mechanism to charge for content access.
-\subsubsection{Formative and Summative Assessment Capabilities}
+\subsubsubsection{Formative and Summative Assessment Capabilities}
LON-CAPA started in 1992 as a system to give personalized homework to students in introductory physics courses. ``Personalized" means that each student sees a different version of the same computer-generated problem: different numbers, choices, graphs, images, simulation parameters, etc, Fig.~\ref{twoproblems}.
\begin{figure}
@@ -670,7 +559,7 @@
Students are generally given immediate feedback on the correctness of their solutions, and in some cases additional help. They are usually granted multiple attempts to get a problem correct. This allows the instructor to follow a learner's thought process, both through statistical analysis (see~\ref{anatool}) and data-mining approaches.
The system also allows for free-form essay-type answers, which are however graded by humans with the assistance of the system (keyword-highlighting, plagiarism-checks, etc).
-\subsubsection{Course Management}
+\subsubsubsection{Course Management}
Over the years, the system added a learning content management system and standard course management features, such as communications, gradebook, etc., which are comparable to commercial course management systems, such as BlackBoard, WebCT, or ANGEL. See
Refs.~\cite{features,edutools} for an overview of features, and comparisons to other systems.
@@ -694,222 +583,44 @@
\end{figure}
-\subsection{Courses}\label{coursesdesc}
-The project will be carried out in the two-semester LBS course sequence LBS 271/272, ``Calculus-Based Introductory Physics I/II." These second-year non-major three-credit courses have a Calculus pre-requisite, and traditionally an enrollment of over 200 students.
-
-Starting Fall 2004, the course will be taught with less total lecturing time, where the third classroom hour will be used for peer-teaching~\cite{mazur} and more frequent quizzes in place of the midterm exams.
-
-Two separate, but associated one-credit laboratory courses (LBS 271L/272L) are required, which most but not all students choose to take simultaneously. Faculty and teaching assistants are frequently assuming shared responsibilities between the lecture and lab courses, with a combined staff of two faculty members and six undergraduate student assistants. The latter are responsible for particular recitation and lab sections, and will be involved in this research project (see Sect.~\ref{undergrad}). Within the duration of this project, the lecture and lab courses might be combined to provide greater coherence between these two venues.
-
-Students in these courses are currently solving approximately 200 online homework problems each semester, most of which currently are of the conventional type.
-\section{\label{sec:method}Methodology}
-
-\section{Research Methodology}\label{analysis}
-\subsection{Establishment of Initial Conditions}
-Many educational studies result in ``no-significant-difference"~\cite{russell}, and particularly the study of question type effectiveness (Sect.~\ref{effect}) may well yield the same result. Many variables may influence the impact of a particular sample of representative problems of a particular type for a particular learner, and it is imperative to understand as much of the ``initial conditions" as possible, since the validity of the hypothesis may depend on them.
-
-\subsubsection{Learner Attitudes, Beliefs, and Expectations}
-Instruments have been developed to assess epistemological beliefs, for example the Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument~\cite{EBAPS}. Related to epistemological beliefs are learnerÕs expectations and attitudes, and of particular interest is the Maryland Physics Expectations (MPEX) survey~\cite{MPEX}.
-\subsubsection{Learner Knowledge about the Topic}\label{prepost}
-We will use existing concept inventory surveys as both pre- and post-tests.
-The qualitative Force Concept Inventory~\cite{fci} and the quantitative companion Mechanical Baseline Test~\cite{hestenesmech} have been used in a large number of studies connected to the teaching of introductory mechanics. The Foundation Coalition has been developing a number of relevant concept inventories~\cite{foundation}, namely the Thermodynamics Concept Inventory, the Dynamics Concept Inventory, and the Electromagnetics Concept Inventory (with two subcomponents, namely Waves and Fields). Since these were designed from an engineering point of view, some adjustment might be necessary. In addition, the Conceptual Survey of Electricity and Magnetism (CSEM)~\cite{maloney} is available for the second semester course.
+\subsection{Computer-Guided Group Formation}
-\subsubsection{Problem Difficulty and Baseline Statistical Data}
-LON-CAPA automatically keeps tracks of the average number of attempts until a problem is solved, as well as the degree of difficulty and the degree of discrimination. This data is cumulative across semesters, and already exists for all assessment problems from their deployment in previous semesters.
+\subsection{Different Question Types}
-\subsection{Observables}
-\subsubsection{Effectiveness}\label{effect}
-Effectiveness will be measured both in terms of performance on summative assessments (quizzes and exams) and on pre-/post-test concept inventory surveys (Sect.~\ref{prepost}). Each item on these instruments will be associated with topically corresponding formative online exercises to determine correlations and differential gain between the feedback types used with the respective online problems. A second posttest, correlated with first semester problems, will be administered at the end of the second semester to determine long-term effects.
+\subsection{Randomized Questions}
-\subsubsection{Problem Solving Technique}
-We intend to focus on a subset of students in the LBS Collaborative Learning Laboratory, and observe them while solving problems. Schoenfeld~\cite{schoenfeld} and Foster~\cite{foster} developed instruments to categorize and document the stages and expertlike
-characteristics~\cite{chi} of observed problem-solving activity by learners, as well as application of metacognitive skills.
-In addition, for all students, log data will be analyzed. Kotas~\cite{kotas} and Minaei~\cite{minaei} developed a mechanism for this log data analysis, which include submission times between attempts, and quality of the entered input.
-\subsubsection{Help-Seeking Behavior and Discussions}\label{discussion}
-It is impossible to observe all help-seeking, but interactions in several settings can be analyzed:
+\section{Dissemination}
+We also invested a great deal of effort disseminating our findings nationwide, as we feel that it is
+crucial to share the results of our research. In the last several years, Eric Mazur and other members of the
+group have given more than one hundred invited talks on peer-instruction in a variety of venues:
+\begin{itemize}
+\item Physics department colloquia at a wide range of institutions from large state universities to small
+liberal arts colleges and community colleges;
+\item Workshops for new faculty sponsored by the American Association of Physics Teachers and the NSFfunded
+Engineering Education Scholars program
+\end{itemize}
+We will present papers at conferences such as the LON-CAPA User Conference, IEEE Frontiers in Education, Educause/NLII, Sloan C, the European Workshop for Multimedia in Physics Education, the Conference on Computer Based Learning in Science (Dr. Kortemeyer presented at these conferences before), the annual meetings of the Deutsche Physikalische Gesellschaft and the Gesellschaft f\"ur Didaktik der Chemie und
+Physik, and the American Association of Physics Teachers Annual and PERC Meetings. We will submit papers to journals such as The Physics Teacher, the American Journal of Physics, Computers and
+Education, and the Journal of Asynchronous Learning Networks. Finally, any content material adapted and implemented in this project will be immediately available to all participating LON-CAPA institutions, and via the LON-CAPA gateway to the NSF-funded National Science Digital Library. Any mature additional platform functionality will be made available in the production releases of the open-source
+freeware LON-CAPA system.
-Online discussions and email communication are preserved within LON-CAPA and can be analyzed even in retrospect for past semesters with respect to relevant behavioral patterns. Table~\ref{discussionex} shows excerpts of online discussions around the two problems in Fig.~\ref{trajectory}.
+\section{Timeline}
\begin{table}
-\tiny
-\begin{tabular}{p{3.1in}|p{3.1in}}
-{\bf Student A:}
-since your not given the initial velocity or the angle,
-but you know the distance covered, couldnt the angle be
-anything as long as the velocity is big enough?
-
-{\bf Student B:}
-The angle could be anything if there was no time given, but
-since there is time given, only one path can be the right
-one.
-
-To solve this problem, you have to take apart the initial
-shot (velocity) into its x- and y-components. Since you
-know the horizontal distance and that air-resistance is
-negligible, the horizontal acceleration is zero (horizontal
-velocity is constant). Hence, you can use the \verb!x = x0 + v0*t + .5*a*t^2! equation to come up with the x-component of
-the initial velocity.
-
-Do the same thing for the y-component: use the equation \verb!y = y0 + v0*t - .5*g*t^2!
-
-Now you have both components of the initial velocity. Put
-these components into a triangle (and use tangent) to get
-the angle, and keep the triangle for the initial velocity
-(hypotenuse).
-
-For the third part, use the y-component of the initial
-velocity in the equation \verb!v^2 - v0^2 = -2*g*(x - h)!, where v
-is the y-component of the velocity at the tip of the arc
-path (...therefore, equals z...), v0 is the y-component of
-the initial velocity, x is the height to find, and h is the
-initial height (a.k.a. x0; it's given).
-
-
-{\bf Student C:}
-How do we use \verb!y = y0 + v0*t - .5*g*t^2!, when we dont have
-two of the variables (y and v0)? How do we use that
-formula to get the vo in the y direction? (i.e. what
-numbers and such do we use?) Thanks.
-
-
-{\bf Student D:}
-Ok. Someone tell me what I'm doing wrong. I figure since
-they give you the distance traveled in the x direction, and
-the time it was in the air, you should be able to get the x
-component of velocity with distance/time.
-
-Now for the y component. My logic was that at half the
-total airtime, the object would be at the peak of the arc,
-and thus would be moving at 0m/s, (being in transition from
-going up to coming down). I tried solving for the initial y
-velocity using this information and the Vfinal=Vinitial +
-(A)(t)equation. But still no luck. Any pointers would be
-greatly appreciated.
-
-{\bf Student E:}
-Hey EVERYBODY, whoever did this FORGOT to divide whatever
-your total change in x is by 2 and use that as the
-displacement in x to find the V nought x, just a little
-heads up
-
-i.e. i used 175 m/ 2 = 87.5 m (since the object launced
-isnt an even parabolic function, its not all of the upside
-down U shape on the graph) as my displacement for finding
-V nought x
-
-{\bf Student F:}
-When using the equation \verb!y = y0 + v0*t - .5*g*t^2!
-the v0 in the second term on the right side is really the
-initial velocity in the y direction not the total initial
-velocity.
-In general the equation \verb!x = x0 + v0*t + .5*a*t^2! is always
-a one demensional equation so when you use it in the y-
-direction all variables are for the y-direction only
-initial y, initial y velocity, and constant y acceleration
-
-&
-
-{\bf Student A:} What does the magnitude of the gravitational field mean?
-
-{\bf Student B:} i'm guessing acceleration
-
-{\bf Teaching Assistant:} That is correct. You need to calculate 'g' for Planet X.
-
-{\bf Student C:}
-How are you supposed to do this problem? I am confused, it
-seems like we have learned nothing during lecture to help
-us understand these problems, we never do any examples and
-work thru problems in lecture. please help.
-
-{\bf Student D:} Yeah, I'm totally lost on this one and all I have to look
-at in my notes are a bunch of variables in an equation, I
-don't know where to plug in half of the numbers I have.
-This problem and the catapult one I'm totally lost on cuz
-all I have to go by are these equations with like 5-6
-variables such as \verb!y=(tan[!$\ddot{y}$\verb!o]*x-(g*x^2/2*(vocos[!$\ddot{y}$\verb!o])^2)! and
-then all the problem tells me is "you threw the rock at
-22.8 m/s" or something...
-
-I got all the other problems done easily, but this one and
-the catapult one... I dunno, I just can't figure them out.
-I worked on them for a while the other day and then got up
-at like 8:30 today to work on them and still haven't
-figured them out. :/
-
-{\bf Student E:} Here is a simple answer to the question, go to sample
-problem 4-7 in your book and you'll get the answer. But
-I'll be nice enough to help you out a little more.
-
-1.) Lecture we talked about getting the tangent line in
-order to find the angle, DO THIS!!! Print out the paper
-and find the angle, IT'S THE ONLY WAY!!!
-
-2.) Sort of kind of eye ball the total distance the
-object traveled from start to finish.
-
-3.) In sample problem 4-7 in the book they used the
-Horizontal Range equation in order to find the answer, but
-you have to adjust the problem to find Gravity or G.
-
-Here is the ADJUSTED equation so all you have to do is
-plug in the numbers that you got.
-
-\verb!G = Intial speed * sin(2*your angle) / Total distance^2!
-
-Now the computer gives you some lead way due to the "eye-
-balling" you have to do, but it gave me my answer and I
-was 0.08 off.
-
-Hope this helps you guy's.
-
-{\bf Student F:}
-what are the units used for this?
-
-{\bf Student G:}
-Gravity is acceleration, so the units should be \verb!m/s^2!.
-
-
-{\bf Student H:}
-Once you plot your points how does this determine your
-angle?
-
-{\bf Student I:}
-I had to do 3 iterations of this problem before getting it
-right. Assuming the math is done correctly there is not
-much tolerance in this problem in regards to calculating
-the launch angle (theta). When I was off by more than 3
-degrees I got it wrong. Be VERY careful when drawing the
-tangent.
+\caption{Proposed timeline by year and institution\label{timeline}}
+\begin{tabular}{|r|l|l|l|}
+\hline
+Year&Harvard&MSU&Erskine\\
+Year 1&\\\hline
+Year 2&\\\hline
+Year 3&\\\hline
\end{tabular}
-\caption{Excerpts from online discussion around the two problems Fig.~\ref{trajectory}\label{discussionex}}
\end{table}
-Discussion contributions and states can be linked to online transactions, such as submission of homework attempts, browsing of content material, and hint usage. Wallace~\cite{wallace} reviewed existing research on such online interactions, however, some adaptation of several of the existing conceptualizations will be necessary to account for the nature of physics courses.
-
-For the subset of LBS students who come to the LBS Collaborative Learning Laboratory, group discussions can to be documented, and linked to online behavior as absolute timing and learner identify are preserved.
-
-Minaei~\cite{minaei} developed data mining strategies to categorize learner behavior, including navigational patterns between assessment and content material, the use of feedback, and communication functions.
-
-Self-reporting can be used for several other help-seeking mechanisms, such as textbook use and peer-interaction~\cite{riffell1,riffell2}.
-
-\subsubsection{Interviews}
-We will interview focus groups of students regarding their experiences and perceived relative helpfulness of the different problem types, and ask them to also reflect on how they perceived these question types were influencing their problem-solving strategies. Pascarella~\cite{pascarella02} developed some frameworks for these interviews, which can be built upon.
-
-\section{Involvement of Students in Research}
-\subsection{Undergraduate}\label{undergrad}
-Students in the courses will be informed about the goals and methods of this project beyond the requirements of the consent procedures. Groups of students will be given the opportunity to participate in focus groups, for which stipends are provided in the budget. The undergraduate teaching assistants in this course will be given the option to participate in the data collection and analysis efforts of the project. Research findings will be made available to the students.
-\subsection{Graduate}
-The budget includes funds for a graduate student assistantship to assist in the materials development effort, as well as to collect and analyze data, and work on publication and dissemination. It is expected that the student will base his or her doctoral work on this project. The idea of offering a doctoral degree in Science Education has been considered at MSU, yet at this point in time, the degree would be conferred by the Department of Physics and Astronomy. The doctoral committee for the student will be comprised of members of the Department of Physics and Astronomy and the Division of Science and Mathematics Education.
-
-
-\section{Evaluation}
-The LON-CAPA Faculty Advisory Board was formed as part of our NSF ITR grant project. It consists of eight actively teaching faculty members and administrators from a number of colleges on campus of MSU, and meets once every month to both evaluate and advise projects connected to LON-CAPA. We propose to continue using this existing structure to evaluate this projectÕs progress and findings. In addition, Dr.~Kortemeyer's Mentoring Committee, which consists of senior faculty members from both LBS and DSME, will guide and advise the progress of this project.
-
-\section{Dissemination}\label{dissem}
-We will present papers at conferences such as the LON-CAPA User Conference, IEEE Frontiers in Education, Educause/NLII, Sloan C, the European Workshop for Multimedia in Physics Education, the Conference on Computer Based Learning in Science (Dr. Kortemeyer presented at these conferences before), the annual meetings of the Deutsche Physikalische Gesellschaft and the Gesellschaft f\"ur Didaktik der Chemie und Physik, and the American Association of Physics Teachers Annual and PERC Meetings. We will submit papers to journals such as The Physics Teacher, the American Journal of Physics, Computers and Education, and the Journal of Asynchronous Learning Networks. Finally, any content material adapted and implemented in this project will be immediately available to all participating LON-CAPA institutions, and via the LON-CAPA gateway to the NSF-funded National Science Digital Library. Any mature additional platform functionality will be made available in the production releases of the open-source freeware LON-CAPA system.
+\section{Expertise and Responsibilites of the PIs}
\section{Results from Prior NSF Support}\label{results}
--www1115657817--