[LON-CAPA-cvs] cvs: modules /gerd/discussions/paper discussions.tex
www
lon-capa-cvs@mail.lon-capa.org
Wed, 04 Jan 2006 18:38:28 -0000
This is a MIME encoded message
--www1136399908
Content-Type: text/plain
www Wed Jan 4 13:38:28 2006 EDT
Modified files:
/modules/gerd/discussions/paper discussions.tex
Log:
Working through reviewer comments
--www1136399908
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20060104133828.txt"
Index: modules/gerd/discussions/paper/discussions.tex
diff -u modules/gerd/discussions/paper/discussions.tex:1.30 modules/gerd/discussions/paper/discussions.tex:1.31
--- modules/gerd/discussions/paper/discussions.tex:1.30 Tue Jan 3 21:44:44 2006
+++ modules/gerd/discussions/paper/discussions.tex Wed Jan 4 13:38:25 2006
@@ -84,10 +84,10 @@
scale without ``curving," and student collaboration was explicitly encouraged. Homework contributed to less
than 20 percent to the final grade.
-In the first semester of the algebra-based course, a total of 134 online problems with 1367 associated discussion
-contributions were analyzed. For the first and second semester of the calculus-based course,
-215 problems with 1078 discussion contributions and 148 problems with 949
-discussion contributions were analyzed, respectively.
+A total of 134 online problems with 1367 associated discussion
+contributions were analyzed in the first semester of the algebra-based course, as well as
+215 problems with 1078 discussion contributions in the first semester, and 148 problems with 949
+discussion contributions in the second semester of the calculus-based course.
In addition, within the first semester calculus-based course (enrollment: 211 students (82 men, 129 women)), discussion characteristics were correlated to student characteristics.
\section{\label{sec:method}Methodology}
@@ -127,9 +127,10 @@
\item[Estimation problems], also known as ``Fermi Problems," require the student to form a model for a scenario, and make reasonable assumptions. A typical example is ``How many barbers are there in Chicago?" or ``How long will I have to wait to find a parking spot?" Students do need to explain their reasoning.
While students find it initially hard to believe that these problems have anything to do with physics, hardly any expert physicist would deny their significance in learning how to solve problems~\cite{mazur96}.
-\item[Qualitative problems] This type of problems asks students to make judgments about physical scenarios, and in that respect are somewhat similar to ranking problems. While the problems themselves are of the type ``Is this high enough?" or ``Can we safely ignore \ldots?," they often do require at least ``back-of-the-envelope" calculations to to give informed answers. As in the case of estimation problems, students do have to explain their reasoning, but the problem itself is usually more structured, and at least the initial answer is more easily evaluated by a computer.
+\item[Qualitative problems] This type of problem asks students to make judgments about physical scenarios, and in that respect are somewhat similar to ranking problems. While the problems themselves are of the type ``Is this high enough?" or ``Can we safely ignore \ldots?," they often do require at least ``back-of-the-envelope" calculations to give informed answers. As in the case of estimation problems, students have to explain their reasoning, but the problem itself is usually more structured, and at least the initial answer is more easily evaluated by a computer.
\item[Essay problems] These are ``explain why" problems. A certain scenario is presented, and students are asked to explain why it turns out the way it does. Students are not asked to recall a certain law --- it is given to them. Instead, they are asked to discuss its validity.
\end{description}
+All 497 online problems available for this study were classified by the author.
The three courses did not include estimation, qualitative, and essay problems, which cannot be graded automatically within the online system.
Table~\ref{table:problemcat} shows the classification distribution of the online
problems available for this project.
@@ -151,24 +152,24 @@
\end{tabular}
\end{ruledtabular}
\end{table*}
-Of the 497 online problems available for this study, none required context-based reasoning, and none expected
+None of the problems required context-based reasoning or expected
a free-form short textual answer. Approximately 14 percent of the problems required representation translation.
The vast majority of problems were conventional numerical problems, which expect
a numerical answer with associated physical unit.
-In addition, for every problem, its
-difficulty index was computed according to the formula
+The
+difficulty index for each problem was computed according to the formula
\begin{equation*}\label{eqn:diffidx}
\mbox{Difficulty Index}=10\left(1-\frac{N_{\mbox{correct}}}{N_{\mbox{attempts}}}\right)
\end{equation*}
-where $N_{\mbox{correct}}$ is the total number of correct solution in the course, and $N_{\mbox{attempts}}$ is the total number of
+where $N_{\mbox{correct}}$ is the total number of correct solutions of the problem in the course, and $N_{\mbox{attempts}}$ is the total number of
correct and incorrect solution submissions (the system allows multiple attempts to arrive at the correct solution, see
subsection~\ref{subsec:system}). If all submissions were correct, meaning, every student would have solved the problem
correctly on the first attempt, the difficulty index would be 0. If none of the submissions were correct, the index would be 10.
\subsection{\label{subsec:disccat}Discussion Classification}
-Student discussion entries were classified into three types and four features. The four types are
+To perform a quantitative discourse analysis of the online discussions, the student discussion entries were classified into three types and four features. The four types are
\begin{description}
\item[Emotional] - discussion contributions were classified as ``emotional" if they mostly communicated opinions,
complaints, gratitude, feelings, etc. Two subtypes were ``positive" and ``negative."
@@ -179,6 +180,7 @@
\item[Conceptual] - contributions that deal with the underlying concepts of the problem. Two subtypes were
``question" and ``answer."
\end{description}
+
In addition, discussion contributions were classified by the following features:
\begin{description}
\item[Unrelated] - the contribution is not related to the problem.
@@ -188,6 +190,8 @@
\item[Physics] - the contribution deals mostly with the physics aspects of the problem.
\end{description}
Table~\ref{table:examples} shows examples of contributions and their classification. Each combination of subtype and feature forms a ``class'' in the analysis.
+
+This coding scheme has to the author's knowledge not been previously used in literature, but was chosen in correspondence to the observations reported in~\cite{lin,chi,pascarella} to distinguish between desirable and undesirable problem solving strategies. Clearly, instructors would want their students to work on a conceptual physics level, yet oftentimes students categorize problems according to surface features~\cite{chi}, and attempt to proceed in a purely procedural approach ("plug-and-chug") to as quickly as possible arrive at the correct solution~\cite{lin}. Pascarella~\cite{pascarella} reports that online homework tends to affirm students in this undesirable approach.
\begin{table*}
\caption{Examples of discussion contribution types and features.\label{table:examples}}
\begin{ruledtabular}
@@ -256,6 +260,7 @@
\end{tabular}
\end{ruledtabular}
\end{table}
+All 3394 discussion contributions were classified by the author over the course of two months. Eight months later, one chapter of the calculus-based course was independently recoded, with an agreement of NN percent.
Different classes were combined into the following
``superclasses'':
@@ -275,7 +280,7 @@
\begin{figure*}
\begin{quote}
-A bug that has a mass $m_b=4g$ walks from the center to the edge of a disk that is freely turning at 32 rpm. The disk has a mass of $m_d=11g$. If the radius of the disk is $R=29cm$, what is the new rate of spinning in rpm?
+{\tt A bug that has a mass $m_b=4g$ walks from the center to the edge of a disk that is freely turning at 32 rpm. The disk has a mass of $m_d=11g$. If the radius of the disk is $R=29cm$, what is the new rate of spinning in rpm?}
\end{quote}
\scriptsize
\begin{tabular}{p{9.3cm}|p{8cm}}
@@ -447,7 +452,7 @@
\end{tabular}
\end{ruledtabular}
\end{table}
-Within the first semester calculus-based course, an analysis by student was performed. Table~\ref{table:disccatfirst} shows the
+Within the first semester calculus-based course, an analysis by student characteristics was performed. Table~\ref{table:disccatfirst} shows the
equivalent of Table~\ref{table:disccat} for this subset of the data. Out of the 211 students who completed the course,
138 students (65 percent) contributed at least one discussion posting over the course of the semester. Figure~\ref{fig:contribBinned} shows the distribution
of number of discussion contributions over the course of the semester. Most students who participated made between one and ten contributions, but one student made
@@ -467,7 +472,7 @@
While the {\it number} of postings is uncorrelated to course grade, their {\it classification}
(subsection~\ref{subsec:disccat}) is correlated.
-In this analysis, the percentage prominance of certain classes
+In this analysis, the percentage of prominance of certain classes
in students' cummulative contributions over the semester was analyzed. The individual percentage (relative) prominances were then averaged by grade.
Note that the outcome is independent of the absolute number of postings a student made, e.g., the discussion behaviour of the student who made 66 contributions is weighed
equally to that of a student having made only the average 5 contributions. Figure~\ref{fig:gradecorrel}
@@ -483,7 +488,8 @@
At the same time, the results confirm that conceptual and physics-related discussions are positively correlated with success in the course, while solution-oriented discussion contributions are strongly negatively correlated. While cause and effect may be arguable, in the following
section~\ref{sec:question}, particular attention needs to be paid to problem properties that elicit either the desirable or undesirable discussion behavioral patterns.
-
+
+Due to the smaller sample size, a correlation analysis by the individual "question" and "answer" classes yielded no statistically significant results.
\section{Results of Analysis by Problem\label{sec:question}}
\subsection{Influence of Problem Difficulty}
Using the full data set of three courses, each discussion contribution associated with a problem was classified according to
@@ -578,7 +584,7 @@
It should be noted that the earlier study dealt with a relatively small set of
representation-translation problems, some of which involved non-static time-evolving simulations as data-source, while in this study, none of the simulation-based problems were assigned. A future study may need to consider the interpretation of time-evolving
simulations as a separate feature, once that more problems of this type exist in the resource pool.
-\subsection{Influence of course}
+\subsection{Influence of the course}
Few significant differences could be found between the algebra-based and the calculus-based course:
\begin{itemize}
\item discussions in the algebra-based course had a significantly higher emotional
@@ -590,6 +596,9 @@
Especially the last observation is discouraging, since as the students in the calculus-based course progressed further into their study of physics, the degree to which they were discussing concepts
decreased. This might partly be due to the different subject matter (electricity and magnetism versus mechanics), but also due to the lack of reward for conceptual considerations in solving standard
homework problems~\cite{lin}.
+
+Again, due to the smaller sample size, a correlation analysis by the individual "question" and "answer" classes yielded no statistically significant results.
+
\subsection{Qualitative Observations}
Reading the online discussions associated with the homework provides valuable insights to the instructor, which are hard to quantify.
When assigning homework, instructors usually have an instructional goal in mind, for example, they would like the students to grapple with a certain concept or work through a specific strategy of problem
@@ -604,16 +613,15 @@
As an example, consider the example Figure~\ref{fig:discussionexample}: there is no external torque, and the problem was meant as a simple example of angular momentum conservation. Since the disk has several centimeters radius, a bug can safely be approximated as a point mass. It is $(\frac{1}{2}m_dr^2+m_b0^2)\omega_0=(\frac{1}{2}m_dr^2+m_br^2)\omega$, and therefore $\omega=\omega_0m_d/(m_d+2m_b)$. As long as the disk is much larger than the bug, the result is independent of its radius, and no unit conversions are needed. Several things jump out to the expert reader of the discussion:
\begin{itemize}
-\item No student mentions the fact that there is no external torque.
-\item No student gives the final solution.
+\item No student mentions the fact that there is no external torque or explicitly mentions angular momentum conservation as the starting point for their considerations.
\item The idea that a bug could be approximated as a ``point mass'' compared to the size of the disk is never mentioned, even though Student E raises the issue.
\item Regarding the calculation of the moment of inertia, there is confusion between the radius of an extended symmetrical object and the radius of the orbit of a point mass (thus, presumably, the question ``what is the radius of the bug?'').
\item Students are plugging in numbers early and do not eliminate the radius of the disk from their calculations (with the possible exception of Student B who hints that ``cancel out some of the things
that are found on both sides of the equation to get a
-better equation that has less numbers in it.'').
+better equation that has less numbers in it.'').
\item Students do not appear to realize that unit conversions are in fact not needed.
\item No student simply posts the final symbolic solution, which is true for virtually all analyzed discussions.
-\item Students went through considerable effort to solve this rather straightforward problem and do not realize that the solution is much simpler to achieve. Here, numerical online homework clearly falls short of handgraded homework, since the students are only graded on the correct final solution, not on their solution strategy.
+\item Students went through considerable effort to solve this rather straightforward problem and do not realize that the solution is much simpler to achieve. Note in particular Student H's comment that ``so many little things can go wrong." Here, numerical online homework clearly falls short of handgraded homework, since the students are only graded on the correct final solution, not on their solution strategy.
\end{itemize}
Particularly the last point is distressing, since it instills a false sense of mastery among the students and confirms them in their undesirable techniques, which is an observation already pointed out by Pascarella~\cite{pascarella} in an earlier study of online homework systems.
The discussion in Figure~\ref{fig:discussionexample} is typical, in spite of the fact that in lecture, problem solving strategies had been discussed, and examples had been given how the derivation of a final result in symbolic form can lead to faster and more reliable results. When discussing examples during lectures, the instructor attempted to model good problem solving strategies.
--www1136399908--