[LON-CAPA-cvs] cvs: modules /gerd/discussions/paper discussions.tex
www
lon-capa-cvs@mail.lon-capa.org
Tue, 03 Jan 2006 18:27:59 -0000
This is a MIME encoded message
--www1136312879
Content-Type: text/plain
www Tue Jan 3 13:27:59 2006 EDT
Modified files:
/modules/gerd/discussions/paper discussions.tex
Log:
Disambiguate "problem" and "question"
--www1136312879
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20060103132759.txt"
Index: modules/gerd/discussions/paper/discussions.tex
diff -u modules/gerd/discussions/paper/discussions.tex:1.28 modules/gerd/discussions/paper/discussions.tex:1.29
--- modules/gerd/discussions/paper/discussions.tex:1.28 Fri Dec 16 16:07:36 2005
+++ modules/gerd/discussions/paper/discussions.tex Tue Jan 3 13:27:56 2006
@@ -45,8 +45,8 @@
online system where the threaded discussion forums are directly attached to randomizing online problems, and in spite of supporting research
(e.g.,~\cite{wallace} for a review) are continually surprised by the
richness of the ensuing peer-interactions. In this study, we are attempting to systematically analyze the student discussion contributions,
-in particular with respect to properties of the courses, the students, and the questions. Our goal is to first identify online discussion
-behavioral patterns of successful students, and in a next step identify the question properties which elicit them.
+in particular with respect to properties of the courses, the students, and the problems. Our goal is to first identify online discussion
+behavioral patterns of successful students, and in a next step identify the problem properties which elicit them.
\subsection{\label{subsec:system}The LON-CAPA Online System}
LON-CAPA started in 1992 as a system to give randomized homework to students in introductory physics courses.
@@ -96,14 +96,14 @@
Kashy~\cite{kashyd01} showed that student mastery of different types of homework problems correlates differently with the students' performance on final exams ---
with multiple-choice non-numerical problems having the lowest correlation, and numerical/mathematical problems that require a translation of representation having the highest.
Steinberg~\cite{steinberg} also analyzed student performance on multiple-choice diagnostics and open-ended exam problems, and found that while those correlate in general, for certain students
-and certain questions, responses differ greatly.
-For this project, we chose a finer-grained classification scheme of question types: Redish~\cite{redish} identifies eight classes and features of exam and homework questions,
+and certain problems, responses differ greatly.
+For this project, we chose a finer-grained classification scheme of problem types: Redish~\cite{redish} identifies eight classes and features of exam and homework problems,
and an adapted version of this scheme will be used:
\begin{description}
-\item[Multiple-choice and short-answer questions] The most basic and most easily computer-evaluated type of question, representing the conventional (typical back-of-chapter textbook) problem.
+\item[Multiple-choice and short-answer problems] The most basic and most easily computer-evaluated type of problem, representing the conventional (typical back-of-chapter textbook) problem.
For the purposes of this project, ``multiple choice" and ``short-answer" will be considered as separate classes, where short-answer includes numerical answers such as ``\verb!17 kg/m^3!," and formula answers, such as ``\verb!1/2*m*(vx^2+vy^2)!." The problems on the left side of Figs.~\ref{threemasses} and \ref{trajectory} are examples of ``short-(numerical)-answer" problems.
-\item[Multiple-choice multiple-response questions] This type of problem, a first step beyond conventional problems, requires a student to evaluate each statement and make a decision about it. The problem on the right side of Fig.~\ref{threemasses} is of this type.
+\item[Multiple-choice multiple-response problems] This type of problem, a first step beyond conventional problems, requires a student to evaluate each statement and make a decision about it. The problem on the right side of Fig.~\ref{threemasses} is of this type.
\begin{figure*}
@@ -111,7 +111,7 @@
\caption{Example of two LON-CAPA problems addressing the same concepts. The problem on the left is a conventional short-numerical-answer problem, while the problem on the right is of type ``multiple-choice multiple-response."\label{threemasses}}
\end{figure*}
-\item[Representation-translation questions] This type of problem requires a student to translate between different representations of the same situation, for example from a graphical to a numerical or textual representation. The answer might be required in different formats, for example in the problem on the right side of Fig.~\ref{trajectory}, it is a short-numerical-answer. Translation between representations can be surprisingly challenging for physics learners~\cite{mcdermott,beichner}.
+\item[Representation-translation problems] This type of problem requires a student to translate between different representations of the same situation, for example from a graphical to a numerical or textual representation. The answer might be required in different formats, for example in the problem on the right side of Fig.~\ref{trajectory}, it is a short-numerical-answer. Translation between representations can be surprisingly challenging for physics learners~\cite{mcdermott,beichner}.
For the purposes of this project, ``representation-translation" will be considered a feature, which may or may not apply to any of the other problem types.
@@ -126,18 +126,18 @@
As in the case of ``representation-translation," ``context-based-reasoning" in this project will be considered a feature, which may apply or may not apply to any of the other problem types.
\item[Estimation problems], also known as ``Fermi Problems," require the student to form a model for a scenario, and make reasonable assumptions. A typical example is ``How many barbers are there in Chicago?" or ``How long will I have to wait to find a parking spot?" Students do need to explain their reasoning.
-While students find it initially hard to believe that these questions have anything to do with physics, hardly any expert physicist would deny their significance in learning how to solve problems~\cite{mazur96}.
-\item[Qualitative questions] This type of questions asks students to make judgments about physical scenarios, and in that respect are somewhat similar to ranking questions. While the questions themselves are of the type ``Is this high enough?" or ``Can we safely ignore \ldots?," they often do require at least ``back-of-the-envelope" calculations to to give informed answers. As in the case of estimation problems, students do have to explain their reasoning, but the question itself is usually more structured, and at least the initial answer is more easily evaluated by a computer.
-\item[Essay questions] These are ``explain why" questions. A certain scenario is presented, and students are asked to explain why it turns out the way it does. Students are not asked to recall a certain law --- it is given to them. Instead, they are asked to discuss its validity.
+While students find it initially hard to believe that these problems have anything to do with physics, hardly any expert physicist would deny their significance in learning how to solve problems~\cite{mazur96}.
+\item[Qualitative problems] This type of problems asks students to make judgments about physical scenarios, and in that respect are somewhat similar to ranking problems. While the problems themselves are of the type ``Is this high enough?" or ``Can we safely ignore \ldots?," they often do require at least ``back-of-the-envelope" calculations to to give informed answers. As in the case of estimation problems, students do have to explain their reasoning, but the problem itself is usually more structured, and at least the initial answer is more easily evaluated by a computer.
+\item[Essay problems] These are ``explain why" problems. A certain scenario is presented, and students are asked to explain why it turns out the way it does. Students are not asked to recall a certain law --- it is given to them. Instead, they are asked to discuss its validity.
\end{description}
The three courses did not include estimation, qualitative, and essay problems, which cannot be graded automatically within the online system.
Table~\ref{table:problemcat} shows the classification distribution of the online
problems available for this project.
\begin{table*}
-\caption{Classification of the online questions according the classification scheme described in
+\caption{Classification of the online problems according the classification scheme described in
subsection~\ref{subsec:problemcat} (adapted from Redish~\cite{redish}). The columns denote the
-different question types, while the rows denote the features of required representation translation and
+different problem types, while the rows denote the features of required representation translation and
context-based reasoning.\label{table:problemcat}}
\begin{ruledtabular}
\begin{tabular}{lccccccc|l}
@@ -151,12 +151,12 @@
\end{tabular}
\end{ruledtabular}
\end{table*}
-Of the 497 online questions available for this study, none required context-based reasoning, and none expected
-a free-form short textual answer. Approximately 14 percent of the questions required representation translation.
-The vast majority of questions were conventional numerical problems, which expect
+Of the 497 online problems available for this study, none required context-based reasoning, and none expected
+a free-form short textual answer. Approximately 14 percent of the problems required representation translation.
+The vast majority of problems were conventional numerical problems, which expect
a numerical answer with associated physical unit.
-In addition, for every question, its
+In addition, for every problem, its
difficulty index was computed according to the formula
\begin{equation*}\label{eqn:diffidx}
\mbox{Difficulty Index}=10\left(1-\frac{N_{\mbox{correct}}}{N_{\mbox{attempts}}}\right)
@@ -430,7 +430,7 @@
\end{figure*}
\begin{table}
-\caption{Same as Table~\ref{table:disccat} for the first semester calculus-based class only. The table includes a small number of contributions by students who eventually dropped the course, which were included in the analysis by question type, but not in the analysis by student characteristics.\label{table:disccatfirst}}
+\caption{Same as Table~\ref{table:disccat} for the first semester calculus-based class only. The table includes a small number of contributions by students who eventually dropped the course, which were included in the analysis by problem type, but not in the analysis by student characteristics.\label{table:disccatfirst}}
\begin{ruledtabular}
\begin{tabular}{lcccccccc|l}
&\multicolumn{2}{c}{Emotional}
@@ -452,7 +452,7 @@
138 students (65 percent) contributed at least one discussion posting over the course of the semester. Figure~\ref{fig:contribBinned} shows the distribution
of number of discussion contributions over the course of the semester. Most students who participated made between one and ten contributions, but one student made
66 postings.
-It is not possible to find out which percentage students {\it read} the discussions, since it is automatically attached to the questions and always visible.
+It is not possible to find out which percentage students {\it read} the discussions, since it is automatically attached to the problems and always visible.
The average number of postings per student was $5\pm0.7$;
female students contributed an average of $5.9\pm1$ postings, while male students contributed an average of $3.7\pm0.7$ postings.
\subsection{Grade-Dependence of Discussion Contributions\label{subsec:gradedep}}
@@ -481,18 +481,18 @@
of the classification approach.
At the same time, the results confirm that conceptual and physics-related discussions are positively correlated with success in the course, while solution-oriented discussion contributions are strongly negatively correlated. While cause and effect may be arguable, in the following
-section~\ref{sec:question}, particular attention needs to be paid to question properties that elicit either the desirable or undesirable discussion behavioral patterns.
+section~\ref{sec:question}, particular attention needs to be paid to problem properties that elicit either the desirable or undesirable discussion behavioral patterns.
-\section{Results of Analysis by Question\label{sec:question}}
-\subsection{Influence of Question Difficulty}
-Using the full data set of three courses, each discussion contribution associated with a question was classified according to
+\section{Results of Analysis by Problem\label{sec:question}}
+\subsection{Influence of Problem Difficulty}
+Using the full data set of three courses, each discussion contribution associated with a problem was classified according to
subsection~\ref{subsec:disccat}. As a measure of the prominence of a class in a given discussion,
the number of contributions belonging to it is divided by the total number of contributions. The discussion characteristics of the problems were binned by their
difficulty index and the average percentage plotted in figure~\ref{fig:diff}. Only superclasses are
shown (subsection~\ref{subsec:problemcat}), namely the emotional climate (crosses), as well as all (questions and answers) related
procedural
(triangles) and conceptual (diamonds) contributions. As an example, the plot is to be interpreted in the following way: within the given
-error boundaries, for a question with difficulty index of six, ten percent of the online discussion is conceptual.
+error boundaries, for a problem with difficulty index of six, ten percent of the online discussion is conceptual.
\begin{figure}
\includegraphics[width=92mm]{KortemeyerFig6}% Here is how to import EPS art
\caption{\label{fig:diff}Discussion characteristics as a function of problem difficulty.
@@ -501,12 +501,12 @@
In addition, the data was fit using second order (procedural, long dashes) and third order (emotional climate, short dashes; conceptual, solid) polynomials.
The greatest variation is found in the emotional climate of the discussion. As is to be expected, the climate is mostly positive
-for ``easy" questions, but then remains positive for a fairly wide range of problem difficulties until it becomes negative
-at a difficulty index of 7. Only six questions had a difficulty index of 9, and --- surprisingly --- none of these had
+for ``easy" problems, but then remains positive for a fairly wide range of problem difficulties until it becomes negative
+at a difficulty index of 7. Only six problems had a difficulty index of 9, and --- surprisingly --- none of these had
associated emotional comments.
For difficulty indizes beyond 3, the prominence of conceptual discussions increases. Surprisingly, it also increases for easier
-questions. This may be attributed to students feeling more confident discussing easier problems on a conceptual level, or simply
+problems. This may be attributed to students feeling more confident discussing easier problems on a conceptual level, or simply
in there being less of a need of procedural discussions.
Overall, the prominence of conceptual discussions is disappointingly low, as it varies between 5 and 16 percent.
@@ -514,7 +514,7 @@
and the emotional climate an indicator of ``pain,'' then beyond a difficulty index of 5 a significant increase in ``pain'' results in a non-significant gain.
Across all difficulties, procedural contributions dominate the discussions, with relatively little significant variance around
-the 40 percent mark. The maximum occurs for questions with a difficulty index of 5.
+the 40 percent mark. The maximum occurs for problems with a difficulty index of 5.
In figure~\ref{fig:diffnochat} the same analysis was carried out, but this time excluding all ``chat" contributions
(subsection~\ref{subsec:problemcat}), i.e., only related non-emotional contributions were considered. The relative prominence of procedural and conceptual discussions systematically
@@ -525,14 +525,14 @@
}
\end{figure}
-\subsection{\label{subsec:qtype}Influence of Question Types}
-Using the full data set of three courses, each question was classified according to subsection~\ref{subsec:problemcat}, and each associated discussion entry according to~\ref{subsec:disccat}. As a measure of the prominence of a class in a given discussion,
+\subsection{\label{subsec:qtype}Influence of Problem Types}
+Using the full data set of three courses, each problem was classified according to subsection~\ref{subsec:problemcat}, and each associated discussion entry according to~\ref{subsec:disccat}. As a measure of the prominence of a class in a given discussion,
the number of contributions belonging to it is divided by the total number of contributions.
-Table~\ref{table:qtype} shows the percentage prominence of discussion contributions with a certain type or with certain features in the discussions associated with questions
+Table~\ref{table:qtype} shows the percentage prominence of discussion contributions with a certain type or with certain features in the discussions associated with problems
that are of a certain type or have certain features.
\begin{table*}
-\caption{Influence of question types and features on discussions.
-The values indicate the percentage prominence of the discussion superclasses, types, and features (columns) for discussions associated with questions of a certain
+\caption{Influence of problem types and features on discussions.
+The values indicate the percentage prominence of the discussion superclasses, types, and features (columns) for discussions associated with problems of a certain
type or with certain features (rows). The values in brackets result from an analysis with ``chat'' excluded.\label{table:qtype}}
\begin{ruledtabular}
\begin{tabular}{lcccccc}
@@ -551,23 +551,23 @@
\end{ruledtabular}
\end{table*}
Error boundaries on the emotional climate values are rather large and mostly include zero (neutral), indicating no significant preferences within the limited sample.
-Yet, students clearly dislike multiple-choice questions, while they clearly like numerical answer problems. The data also indicates that students prefer ``conventional'' over
+Yet, students clearly dislike multiple-choice problems, while they clearly like numerical answer problems. The data also indicates that students prefer ``conventional'' over
representation-translation problems.
The prominence of procedural discussions is significantly higher for numerical problems than for any other problem types, and higher for ``conventional'' than for
representation-translation problems. The latter difference vanishes when ``chat'' is excluded.
Solution-oriented contributions are significantly higher for multiple-choice and multiple-choice-multiple-response problems than for the other problem types with the exception
-of formula-response questions, where error-boundaries overlap. In spite of the randomization provided, in discussion entries, students frequently reverse-engineered the complete randomization space by copying their correct answer screens into the discussions
+of formula-response problems, where error-boundaries overlap. In spite of the randomization provided, in discussion entries, students frequently reverse-engineered the complete randomization space by copying their correct answer screens into the discussions
(see the example for a surface-level solution-oriented discussion entry in Table~\ref{table:examples}).
-The prominence of mathematical discussion contributions is the highest for formula-response questions, approximately equal for numerical and single-response multiple-choice questions, and the lowest for multiple-choice-multiple-response, ranking, and click-on-image questions.
+The prominence of mathematical discussion contributions is the highest for formula-response problems, approximately equal for numerical and single-response multiple-choice problems, and the lowest for multiple-choice-multiple-response, ranking, and click-on-image problems.
-The prominence of physics-related discussion contributions was the highest for ranking and click-on-image problems, and the lowest for multiple-choice questions.
+The prominence of physics-related discussion contributions was the highest for ranking and click-on-image problems, and the lowest for multiple-choice problems.
Finally, when it comes to conceptual discussions, their prominence is significantly lower in single-response multiple-choice and numerical problems than in the other problem types. In the
-earlier study by Kashy~\cite{kashyd01}, it was also found that mastery of these same question types does not predict overall performance on the final exam as well as other question types.
-Multiple-choice problems that do not involve numbers are frequently called ``conceptual'' questions, but in this study, it was found that they do not necessarily lead to conceptual discussions.
+earlier study by Kashy~\cite{kashyd01}, it was also found that mastery of these same problem types does not predict overall performance on the final exam as well as other problem types.
+Multiple-choice problems that do not involve numbers are frequently called ``conceptual'' problems, but in this study, it was found that they do not necessarily lead to conceptual discussions.
It is a surprising result that the only significant difference between ``conventional'' and representation-translation problems is that students discuss slightly less procedure in favor of
more complaints, and that differences disappear when ``chat'' is excluded from the analysis. McDermott~\cite{mcdermott} and Beichner~\cite{beichner} on the other hand found that students have unexpected difficulties in translating for example data presented as graphs, so a stronger effect of this feature was expected. In additon, Kashy~\cite{kashyd01} found that mastery of representation-translation problems
@@ -630,13 +630,13 @@
Online student discussions are a rich source of insight into student problem solving behavior. It was verified that indeed conceptual and physics-related discussion contributions are characteristics of students who are successful in the course, while the prominance of solution-oriented
discussion contributions is strongly negatively correlated with success in the course.
-Different discussion patterns ensue around different question characteristics:
+Different discussion patterns ensue around different problem characteristics:
\begin{description}
\item[Difficulty] Very easy problems can elicit a high level conceptual discussions,
and so can problems of mid-range difficulty. As problems become more difficult, there is no significant gain in conceptual discussions.
-\item[Question Types] Different question types result in different association discussion patterns. Discussions on a procedural level are more prominent for numerical problems than for any other problem type. Solution-oriented discussions are more prominent for multiple-choice style questions in an effort to short-circuit the conceptual reasoning.
-Discussions around single-response multiple choice questions and numerical questions have a significantly lower prominance of conceptual discussions than other question types.
-Ranking questions show very favorable discussion patterns, but their sample size has been too small to make definitive statements.
+\item[Problem Types] Different problem types result in different association discussion patterns. Discussions on a procedural level are more prominent for numerical problems than for any other problem type. Solution-oriented discussions are more prominent for multiple-choice style problems in an effort to short-circuit the conceptual reasoning.
+Discussions around single-response multiple choice problems and numerical problems have a significantly lower prominance of conceptual discussions than other problem types.
+Ranking problems show very favorable discussion patterns, but their sample size has been too small to make definitive statements.
\end{description}
Analyzing online discussions around problems has been found to provide valuable insights into student problem-solving strategies.
\begin{acknowledgments}
--www1136312879--