[LON-CAPA-cvs] cvs: modules /gerd/discussions studeval.pl /gerd/discussions/paper discussions.bib discussions.tex
www
lon-capa-cvs@mail.lon-capa.org
Tue, 26 Apr 2005 13:59:44 -0000
This is a MIME encoded message
--www1114523984
Content-Type: text/plain
www Tue Apr 26 09:59:44 2005 EDT
Modified files:
/modules/gerd/discussions studeval.pl
/modules/gerd/discussions/paper discussions.bib discussions.tex
Log:
Updates
--www1114523984
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20050426095944.txt"
Index: modules/gerd/discussions/studeval.pl
diff -u modules/gerd/discussions/studeval.pl:1.6 modules/gerd/discussions/studeval.pl:1.7
--- modules/gerd/discussions/studeval.pl:1.6 Thu Apr 14 16:10:28 2005
+++ modules/gerd/discussions/studeval.pl Tue Apr 26 09:59:44 2005
@@ -80,6 +80,7 @@
my %statn=();
my %statsumsquare=();
+my @sumgrades=();
my @bins=0;
#
# Compute means and errors
@@ -88,7 +89,10 @@
foreach my $error (0,1) {
foreach (@q) {
my @entries=split(/\,/,$_);
- unless ($error) { $bins[$entries[$sumall]]++; }
+ unless ($error) {
+ $bins[$entries[$sumall]]++;
+ $sumgrades[$entries[$sumall]]+=$entries[$grade];
+ }
&callabssums('sumall&all',$error,@entries);
&callabssums('sumall&gender'.$entries[$gender],$error,@entries);
&callabssums('sumall&grade'.$entries[$grade],$error,@entries);
@@ -124,7 +128,11 @@
print "\n";
for (my $i=0; $i<=$#bins; $i++) {
- print "\n".$i.','.$bins[$i];
+ my $avegrade='';
+ if ($bins[$i]) {
+ $avegrade=$sumgrades[$i]/$bins[$i];
+ }
+ print "\n".$i.','.$bins[$i].','.$avegrade;
}
print "\n";
Index: modules/gerd/discussions/paper/discussions.bib
diff -u modules/gerd/discussions/paper/discussions.bib:1.5 modules/gerd/discussions/paper/discussions.bib:1.6
--- modules/gerd/discussions/paper/discussions.bib:1.5 Fri Apr 22 17:29:12 2005
+++ modules/gerd/discussions/paper/discussions.bib Tue Apr 26 09:59:44 2005
@@ -7,10 +7,10 @@
title = "The problem with problems"
}
-@ARTICLE{mazur97,
+@BOOK{mazur97,
author = "Eric Mazur",
year = "1997",
- booktitle = "Peer Instruction",
+ title = "Peer Instruction",
isbn="0-13-565441-6",
publisher="Prentice Hall"
}
@@ -43,26 +43,26 @@
}
-@ARTICLE{redish,
+@BOOK{redish,
author = "Edward F. (Joe) Redish",
- booktitle = "Teaching Physics",
+ title = "Teaching Physics",
isbn="0-471-39378-9",
publisher="Wiley",
year = "2003",
}
-@ARTICLE{jitt,
+@BOOK{jitt,
author = "Gregor M. Novak and Evelyn T. Patterson and Andrew D. Gavrin and Wolfgang Christian",
- booktitle = "Just-in-time teaching: blending active learning with web technology",
+ title = "Just-in-time teaching: blending active learning with web technology",
isbn="0-13-085034-9",
publisher="Prentice Hall",
year = "1999",
}
-@ARTICLE{kashyd01,
+@CONFERENCE{kashyd01,
author = "Deborah A. Kashy and Guy Albertelli and Guy Ashkenazi and Edwin Kashy and Hon-Kie Ng and Michael Thoennessen",
year = "2001",
- journal = "Proc. Frontiers in Education",
+ booktitle = "Proc. Frontiers in Education",
volume = "31",
pages = "1073-1078",
isbn = "0-7803-6669-7",
Index: modules/gerd/discussions/paper/discussions.tex
diff -u modules/gerd/discussions/paper/discussions.tex:1.16 modules/gerd/discussions/paper/discussions.tex:1.17
--- modules/gerd/discussions/paper/discussions.tex:1.16 Mon Apr 25 11:43:10 2005
+++ modules/gerd/discussions/paper/discussions.tex Tue Apr 26 09:59:44 2005
@@ -50,18 +50,18 @@
\subsection{\label{subsec:system}The LON-CAPA Online System}
LON-CAPA started in 1992 as a system to give randomized homework to students in introductory physics courses.
-``Randomized" means that each student sees a different version of the same computer-generated problem: different numbers, choices, graphs, images, simulation parameters, etc, Fig.~\ref{twoproblems}.
+``Randomized" means that each student sees a different version of the same computer-generated problem: different numbers, choices, graphs, images, simulation parameters, etc, see Fig.~\ref{twoproblems}.
\begin{figure*}
\includegraphics[width=6.5in]{atwood}
\caption{Web-rendering of the same LON-CAPA problem for two different students.\label{twoproblems}
}
\end{figure*}
Randomization was implemented as a means to both control ``cheating" and foster student collaboration on a
-conceptual level --- since problem answers will differ from student to student, they cannot simply exchange the
+conceptual level --- since problem answers will differ from student to student, learners cannot simply exchange the
correct answers when collaborating with each other.
LON-CAPA allows for immediate feedback on problem correctness to the student, as well as multiple tries to arrive at
-the correct solution (both features could be disabled by the instructor). LON-CAPA is designed to foster
+the correct solution (both features could be disabled by the instructor). The system is designed to foster
communication among the learners, and asynchronous threaded discussion boards
are attached directly to the bottom of every online resource.
For the purposes of this project, it is thus possible
@@ -91,16 +91,16 @@
\section{\label{sec:method}Methodology}
\subsection{\label{subsec:problemcat}Problem Classification}
-Kashy~\cite{kashyd01} showed that student mastery of different types of homework problems correlates differently with the student's performance on final exams ---
+Kashy~\cite{kashyd01} showed that student mastery of different types of homework problems correlates differently with the students' performance on final exams ---
with multiple-choice non-numerical problems having the lowest correlation, and numerical/mathematical problems that require a translation of representation having the highest.
Steinberg~\cite{steinberg} also analyzed student performance on multiple-choice diagnostics and open-ended exam problems, and found that while those correlate in general, for certain students
and certain questions, responses differ greatly.
For this project, we chose a finer-grained classification scheme of question types: Redish~\cite{redish} identifies eight classes and features of exam and homework questions,
-an adapted version of which will be used:
+and an adapted version of this scheme will be used:
\begin{description}
\item[Multiple-choice and short-answer questions] The most basic and most easily computer-evaluated type of question, representing the conventional (typical back-of-chapter textbook) problem.
-For the purposes of this project, ``multiple choice" and ``short-answer" will be considered as separate classes, where short-answer includes numerical answers such as ``$17 kg/m^3$," and formula answers, such as ``\verb!1/2*m*(vx^2+vy^2)!." The problems on the left side of Figs.~\ref{threemasses} and \ref{trajectory} are examples of ``short-(numerical)-answer" problems.
+For the purposes of this project, ``multiple choice" and ``short-answer" will be considered as separate classes, where short-answer includes numerical answers such as ``\verb!17 kg/m^3!," and formula answers, such as ``\verb!1/2*m*(vx^2+vy^2)!." The problems on the left side of Figs.~\ref{threemasses} and \ref{trajectory} are examples of ``short-(numerical)-answer" problems.
\item[Multiple-choice multiple-response questions] This type of problem, a first step beyond conventional problems, requires a student to evaluate each statement and make a decision about it. The problem on the right side of Fig.~\ref{threemasses} is of this type.
@@ -109,7 +109,7 @@
\caption{Example of two LON-CAPA problems addressing the same concepts. The problem on the left is a conventional short-numerical-answer problem, while the problem on the right is of type ``multiple-choice multiple-response."\label{threemasses}}
\end{figure*}
-\item[Representation-translation questions] This type of problem requires a student to translate between different representations of the same situation, for example from a graphical to a numerical or textual representation. The answer might be given in different formats, for example in the problem on the right side of Fig.~\ref{trajectory}, it is a short-numerical-answer. Translation between representations can be surprisingly challenging for physics learners~\cite{mcdermott,beichner}.
+\item[Representation-translation questions] This type of problem requires a student to translate between different representations of the same situation, for example from a graphical to a numerical or textual representation. The answer might be required in different formats, for example in the problem on the right side of Fig.~\ref{trajectory}, it is a short-numerical-answer. Translation between representations can be surprisingly challenging for physics learners~\cite{mcdermott,beichner}.
For the purposes of this project, ``representation-translation" will be considered a feature, which may or may not apply to any of the other problem types.
@@ -118,7 +118,7 @@
\caption{Example of two LON-CAPA problems addressing the same concepts in two different representations. The problem on the left is a conventional short-numerical-answer problem, while the problem on the right requires ``representation-translation."\label{trajectory}}
\end{figure*}
-\item[Ranking-tasks] This type of problem requires a student to rank a number of statements, scenarios, or objects with respect to a certain feature. For example, a student might be asked to rank a number of projectiles in the order that they will hit the ground, or a number of points in order of the strength of their local electric potential.
+\item[Ranking-tasks] This type of problem requires a student to rank a number of statements, scenarios, or objects with respect to a certain feature. For example, a student might be asked to rank a number of projectiles in the order that they will hit the ground, or a number of locations in order of the strength of their local electric potential.
\item[Context-based reasoning problems] The distinguishing characteristic of these problems is that they are set in the context of real-world scenarios and not in the context of the artificial ``zero-friction" laboratory scenarios of typical textbook problems.
As in the case of ``representation-translation," ``context-based-reasoning" in this project will be considered a feature, which may apply or may not apply to any of the other problem types.
@@ -153,6 +153,18 @@
a free-form short textual answer. Approximately 14 percent of the questions required representation translation.
The vast majority of questions were conventional numerical problems, which expect
a numerical answer with associated physical unit.
+
+In addition, for every question, its
+difficulty index was computed according to the formula
+\begin{equation*}\label{eqn:diffidx}
+\mbox{Difficulty Index}=10\left(1-\frac{N_{\mbox{correct}}}{N_{\mbox{attempts}}}\right)
+\end{equation*}
+where $N_{\mbox{correct}}$ is the total number of correct solution in the course, and $N_{\mbox{attempts}}$ is the total number of
+correct and incorrect solution submissions (the system allows multiple attempts to arrive at the correct solution, see
+subsection~\ref{subsec:system}). If all submissions were correct, meaning, every student would have solved the problem
+correctly on the first attempt, the difficulty index would be 0. If none of the submissions were correct, the index would be 10.
+
+
\subsection{\label{subsec:disccat}Discussion Classification}
Student discussion entries were classified into three types and four features. The four types are
\begin{description}
@@ -175,9 +187,7 @@
\end{description}
Table~\ref{table:examples} shows examples of contributions and their classification.
\begin{table*}
-\caption{Influence of question types and features on discussions.
-The values indicate the percentage prominence of the discussion superclasses, types, and features (columns) for discussions associated with questions of a certain
-type or with certain features (rows). The values in brackets result from an analysis with ``chat'' excluded.\label{table:examples}}
+\caption{Examples of discussion contribution types and features.\label{table:examples}}
\begin{ruledtabular}
\begin{tabular}{l|p{3.9cm}|p{3.9cm}|p{3.9cm}|p{3.9cm}}
&Unrelated&Solution&Math&Physics\\\hline
@@ -244,25 +254,9 @@
\end{tabular}
\end{ruledtabular}
\end{table}
-The majority of the discussion contributions were of type surface-level or procedural, followed by emotional
-contributions.
-The vast majority of discussion contributions had the feature of being solution-oriented,
-yet a considerable number dealt with the physics
-of the problems.
-\subsection{Computation of Results\label{subsec:comp}}
-Each question was classified according to the types and features described in subsection~\ref{subsec:problemcat}. In addition, its
-difficulty index was computed according to the formula
-\begin{equation*}\label{eqn:diffidx}
-\mbox{Difficulty Index}=10\left(1-\frac{N_{\mbox{correct}}}{N_{\mbox{attempts}}}\right)
-\end{equation*}
-where $N_{\mbox{correct}}$ is the total number of correct solution in the course, and $N_{\mbox{attempts}}$ is the total number of
-correct and incorrect solution submissions (the system allows multiple attempts to arrive at the correct solution, see
-subsection~\ref{subsec:system}). If all submissions were correct, meaning, every student would have solved the problem
-correctly on the first attempt, the difficulty index would be 0. If none of the submissions were correct, the index would be 10.
-Each associated discussion contribution was classified according to the types and features described in
-subsection~\ref{subsec:disccat}. As a measure of the prominence of a class in a given discussion,
-the number of contributions belonging to it is divided by the total number of contributions. In addition, the following
+
+In addition, the following
superclasses are considered:
\begin{description}
\item[Chat] - all contributions that are unrelated or emotional.
@@ -270,6 +264,14 @@
contributions. This number would be negative if the problem led to mostly negative emotional comments.
\item[Type and feature sums] - number of all related contributions belonging to a certain type, subtype, or feature.
\end{description}
+
+The majority of the discussion contributions were of type surface-level or procedural, followed by emotional
+contributions.
+The vast majority of discussion contributions had the feature of being solution-oriented,
+yet a considerable number dealt with the physics
+of the problems.
+
+
\section{Results of Analysis by Student}
\subsection{Participation}
\begin{figure*}
@@ -311,9 +313,12 @@
\section{Results of Analysis by Question\label{sec:question}}
\subsection{Influence of Question Difficulty}
-The discussion characteristics of the problems were binned by their
-difficulty index (equation~\ref{eqn:diffidx}) and the average percentage plotted in figure~\ref{fig:diff}. Only superclasses are
-shown (subsection~\ref{subsec:comp}), namely the emotional climate (crosses), as well as all (questions and answers) related
+Each question was classified according to the types and features described in subsection~\ref{subsec:problemcat}.
+Each associated discussion contribution was classified according to the types and features described in
+subsection~\ref{subsec:disccat}. As a measure of the prominence of a class in a given discussion,
+the number of contributions belonging to it is divided by the total number of contributions. The discussion characteristics of the problems were binned by their
+difficulty index and the average percentage plotted in figure~\ref{fig:diff}. Only superclasses are
+shown (subsection~\ref{subsec:problemcat}), namely the emotional climate (crosses), as well as all (questions and answers) related
procedural
(triangles) and conceptual (diamonds) contributions. As an example, the plot is to be interpreted in the following way: within the given
error boundaries, for a question with difficulty index of six, ten percent of the online discussion is conceptual.
@@ -341,7 +346,7 @@
the 40 percent mark. The maximum occurs for questions with a difficulty index of 5.
In figure~\ref{fig:diffnochat} the same analysis was carried out, but this time excluding all ``chat" contributions
-(subsection~\ref{subsec:comp}), i.e., only related non-emotional contributions were considered. The relative prominence of procedural and conceptual discussions systematically
+(subsection~\ref{subsec:problemcat}), i.e., only related non-emotional contributions were considered. The relative prominence of procedural and conceptual discussions systematically
increases, but all observations from the full analysis remain valid. ``Chat'' mostly provides a constant background across all difficulty indices.
\begin{figure}
\includegraphics[width=92mm]{diffnochat}% Here is how to import EPS art
@@ -364,7 +369,7 @@
Numerical&4$\pm$1&48$\pm$1 (57$\pm$1)&52$\pm$1 (63$\pm$2)&8$\pm$1 (9$\pm$1)&23$\pm$1 (27$\pm$1)&7$\pm$1 (8$\pm$1)\\
Formula&6$\pm$8&29$\pm$11 (31$\pm$10)&57$\pm$16 (64$\pm$18)&31$\pm$16 (36$\pm$18)&&\\
Mult.-choice Mult.-resp.&1$\pm$1&15$\pm$3 (16$\pm$3)&66$\pm$4 (72$\pm$4)&1$\pm$1 (2$\pm$2)&22$\pm$3 (26$\pm$3)&14$\pm$2 (18$\pm$3)\\
-Ranking&2$\pm$3&24$\pm$11 (26$\pm$12)&41$\pm$18 (46$\pm$20)&0$\pm$0 (0$\pm$0)&52$\pm$20 (54$\pm$20)&38$\pm$18 (39$\pm$17)\\
+Ranking&2$\pm$3&24$\pm$11 (26$\pm$12)&41$\pm$18 (46$\pm$20)&&52$\pm$20 (54$\pm$20)&38$\pm$18 (39$\pm$17)\\
Click-on-Image&0$\pm$9&14$\pm$6 (18$\pm$8)&53$\pm$8 (69$\pm$11)&3$\pm$3 (5$\pm$5)&25$\pm$11 (26$\pm$11)&22$\pm$8 (25$\pm$9)\\\hline
``Conventional''&4$\pm$1&42$\pm$1 (50$\pm$2)&55$\pm$1 (65$\pm$2)&7$\pm$1 (8$\pm$1)&23$\pm$1 (27$\pm$1)&9$\pm$1 (10$\pm$1)\\
--www1114523984--