[LON-CAPA-cvs] cvs: modules /gerd/discussions/paper originalsubmission.tex

[www]

This is a MIME encoded message

--www1134268129
Content-Type: text/plain

www		Sat Dec 10 21:28:49 2005 EDT

  Added files:                 
    /modules/gerd/discussions/paper	originalsubmission.tex 
  Log:
  Original submission
  
  
--www1134268129
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20051210212849.txt"


Index: modules/gerd/discussions/paper/originalsubmission.tex
+++ modules/gerd/discussions/paper/originalsubmission.tex
\documentclass[twocolumn,showpacs,preprintnumbers,amsmath,amssymb]{revtex4}
%\documentclass[preprint,showpacs,preprintnumbers,amsmath,amssymb]{revtex4}

% Some other (several out of many) possibilities
%\documentclass[preprint,aps]{revtex4}
%\documentclass[preprint,aps,draft]{revtex4}

\usepackage{graphicx}% Include figure files
\usepackage{dcolumn}% Align table columns on decimal point
\usepackage{bm}% bold math

%\nofiles

\begin{document}

%\preprint{APS/123-QED}

\title{An Analysis of\\Asynchronous Online Homework Discussions\\in Introductory Physics Courses}% Force line breaks with \\

\author{Gerd Kortemeyer}
\email{korte@lite.msu.edu}
\homepage{http://www.lite.msu.edu/kortemeyer/}
\affiliation{
Lyman-Briggs School of Science\\
and Division of Science and Mathematics Education\\
Michigan State University\\
East Lansing, MI 48825
}%

\date{\today}

\begin{abstract}
Asynchronous online student discussion entries around online homework problems in introductory physics courses
are analyzed with respect to course type, student course performance, student gender, problem difficulty, and problem type. It is found that 
these variables can significantly change the character of the online student collaborations. 
\end{abstract}

\pacs{01.40.Fk,01.40.Gm,01.50.Ht,01.50.Kw}

\maketitle

\section{\label{sec:intro}Introduction}
Students discussing physics with their peers in-class has proven to be an effective way of teaching~\cite{mazur97}, and the practice has found 
wide-spread acceptance. Using online forums, the practice can be extended outside the classroom. Over the past years, we have been using an 
online system where the threaded discussion forums are directly attached to randomizing online problems, and in spite of supporting research
(e.g.,~\cite{wallace} for a review) are continually surprised by the 
richness of the ensuing peer-interactions. In this study, we are attempting to systematically analyze the student discussion contributions, 
in particular with respect to properties of the courses, the students, and the questions. Our goal is to first identify online discussion 
behavioral patterns of successful students, and in a next step identify the question properties which elicit them.

\subsection{\label{subsec:system}The LON-CAPA Online System}
LON-CAPA started in 1992 as a system to give randomized homework to students in introductory physics courses.  
``Randomized" means that each student sees a different version of the same computer-generated problem: different numbers, choices, graphs, images, simulation parameters, etc, see Fig.~\ref{twoproblems}.
\begin{figure*}
\includegraphics[width=6.5in]{KortemeyerFig1}
\caption{Web-rendering of the same LON-CAPA problem for two different students.\label{twoproblems}
}
\end{figure*}
Randomization was implemented as a means to both control ``cheating" and foster student collaboration on a 
conceptual level --- since problem answers will differ from student to student, learners cannot simply exchange the 
correct answers when collaborating with each other.

LON-CAPA allows for immediate feedback on problem correctness to the student, as well as multiple tries to arrive at 
the correct solution (both features could be disabled by the instructor). The system is designed to foster 
communication among the learners, and asynchronous threaded discussion boards
are attached directly to the bottom of every online resource.
For the purposes of this project, it is thus possible
to establish a one-to-one association between an online problems and discussions.

Students can post anonymously or using a screenname, however, the full name is always visible to the instructors (students know this). Also, occasionally, instructors are posting to the discussion. 
Over time, competing discussion sites developed outside of LON-CAPA, which are completely anonymous and not visited by instructors. Kashy~\cite{kashy03} found that the use of the internal discussion sites is positively correlated to course grades and FCI scores, while the use of the external sites is negatively correlated to these scores.

In addition, LON-CAPA keeps statistical data for every problem, which allows to associate problems with their 
degree of difficulty.
\subsection{\label{subsec:courses}Courses}
Discussions from three courses at Michigan State University were analyzed, namely, the first semester of
an algebra-based course with students from a wide variety of majors, 
as well as the first and the second semester of a calculus-based course with a majority
of pre-medical students. In both courses, the complete teaching material was provided online, with homework problems
embedded. No textbook was required in either course. 
The algebra-based course had one section that was completely taught online, but the majority of the students 
in the algebra-based course, and all students in the calculus-based course, had regular lectures throughout the week.
In the case of the calculus-based course, a parallel lab was offered. All three courses were graded on an absolute
scale without ``curving," and student collaboration was explicitly encouraged. Homework contributed to less 
than 20 percent to the final grade.

In the first-semester algebra-based course, a total of 134 online problems with 1367 associated discussion 
contributions were analyzed, 
as well as 215 problems with 1078 discussion contributions and 148 problems with 949 
discussion contributions in the first and second semester of the calculus-based course, respectively. 

In addition, within the first semester calculus-based course (enrollment: 211 students (82 men, 129 women)), discussions were analyzed by student.
\section{\label{sec:method}Methodology}

\subsection{\label{subsec:problemcat}Problem Classification}
Kashy~\cite{kashyd01} showed that student mastery of different types of homework problems correlates differently with the students' performance on final exams --- 
with multiple-choice non-numerical problems having the lowest correlation, and numerical/mathematical problems that require a translation of representation having the highest.
Steinberg~\cite{steinberg} also analyzed student performance on multiple-choice diagnostics and open-ended exam problems, and found that while those correlate in general, for certain students
and certain questions, responses differ greatly. 
For this project, we chose a finer-grained classification scheme of question types: Redish~\cite{redish} identifies eight classes and features of exam and homework questions, 
and an adapted version of this scheme will be used:
\begin{description}
\item[Multiple-choice and short-answer questions] The most basic and most easily computer-evaluated type of question, representing the conventional (typical back-of-chapter textbook) problem.

For the purposes of this project, ``multiple choice" and ``short-answer" will be considered as separate classes, where short-answer includes numerical answers such as ``\verb!17 kg/m^3!," and formula answers, such as ``\verb!1/2*m*(vx^2+vy^2)!."  The problems on the left side of Figs.~\ref{threemasses} and \ref{trajectory} are examples of ``short-(numerical)-answer" problems.
\item[Multiple-choice multiple-response questions]  This type of problem, a first step beyond conventional problems, requires a student to evaluate each statement and make a decision about it. The problem on the right side of Fig.~\ref{threemasses} is of this type.


\begin{figure*}
\includegraphics[width=6.5in]{KortemeyerFig2}
\caption{Example of two LON-CAPA problems addressing the same concepts. The problem on the left is a conventional short-numerical-answer problem, while the problem on the right is of type ``multiple-choice multiple-response."\label{threemasses}}
\end{figure*}

\item[Representation-translation questions] This type of problem requires a student to translate between different representations of the same situation, for example from a graphical to a numerical or textual representation. The answer might be required in different formats, for example in the problem on the right side of Fig.~\ref{trajectory}, it is a short-numerical-answer. Translation between representations can be surprisingly challenging for physics learners~\cite{mcdermott,beichner}.

For the purposes of this project, ``representation-translation" will be considered a feature, which may or may not apply to any of the other problem types.

\begin{figure*}
\includegraphics[width=6.5in]{KortemeyerFig3}
\caption{Example of two LON-CAPA problems addressing the same concepts in two different representations. The problem on the left is a conventional short-numerical-answer problem, while the problem on the right requires ``representation-translation."\label{trajectory}}
\end{figure*}

\item[Ranking-tasks] This type of problem requires a student to rank a number of statements, scenarios, or objects with respect to a certain feature. For example, a student might be asked to rank a number of projectiles in the order that they will hit the ground, or a number of locations in order of the strength of their local electric potential.
\item[Context-based reasoning problems] The distinguishing characteristic of these problems is that they are set in the context of real-world scenarios and not in the context of the artificial ``zero-friction" laboratory scenarios of typical textbook problems.

As in the case of ``representation-translation," ``context-based-reasoning" in this project will be considered a feature, which may apply or may not apply to any of the other problem types.
\item[Estimation problems], also known as ``Fermi Problems," require the student to form a model for a scenario, and make reasonable assumptions. A typical example is ``How many barbers are there in Chicago?" or ``How long will I have to wait to find a parking spot?" Students do need to explain their reasoning.

While students find it initially hard to believe that these questions have anything to do with physics, hardly any expert physicist would deny their significance in learning how to solve problems~\cite{mazur96}. 
\item[Qualitative questions] This type of questions asks students to make judgments about physical scenarios, and in that respect are somewhat similar to ranking questions. While the questions themselves are of the type ``Is this high enough?" or ``Can we safely ignore \ldots?," they often do require at least ``back-of-the-envelope" calculations to to give informed answers. As in the case of estimation problems, students do have to explain their reasoning, but the question itself is usually more structured, and at least the initial answer is more easily evaluated by a computer.
\item[Essay questions] These are ``explain why" questions. A certain scenario is presented, and students are asked to explain why it turns out the way it does. Students are not asked to recall a certain law --- it is given to them. Instead, they are asked to discuss its validity.
\end{description}
The three courses did not include estimation, qualitative, and essay problems, which cannot be graded automatically within the online system.
Table~\ref{table:problemcat} shows the classification distribution of the online 
problems available for this project.

\begin{table*}
\caption{Classification of the online questions according the classification scheme described in 
subsection~\ref{subsec:problemcat} (adapted from Redish~\cite{redish}). The columns denote the
different question types, while the rows denote the features of required representation translation and
context-based reasoning.\label{table:problemcat}}
\begin{ruledtabular}
\begin{tabular}{lccccccc|l}
&\multicolumn{4}{c}{Multiple-choice and short-answer}&Mult.-choice mult.-resp.&Ranking&Click-on-image&\\
&Multiple-choice&Tex\-tual&Nume\-rical&For\-mula&&&\\
\hline
``Conventional''&   10&     &  355&    3&   54&    4&   2&428\\
Rep-Trans       &    7&     &   38&     &   16&    1&   7&69\\
Context-based   &     &     &     &     &     &     &    &0\\\hline
                &   17&     &  393&    3&   70&    5&   9&497 
\end{tabular}
\end{ruledtabular}
\end{table*}
Of the 497 online questions available for this study, none required context-based reasoning, and none expected 
a free-form short textual answer. Approximately 14 percent of the questions required representation translation.
The vast majority of questions were conventional numerical problems, which expect
a numerical answer with associated physical unit. 

In addition, for every question, its
difficulty index was computed according to the formula
\begin{equation*}\label{eqn:diffidx}
\mbox{Difficulty Index}=10\left(1-\frac{N_{\mbox{correct}}}{N_{\mbox{attempts}}}\right)
\end{equation*}
where $N_{\mbox{correct}}$ is the total number of correct solution in the course, and $N_{\mbox{attempts}}$ is the total number of
correct and incorrect solution submissions (the system allows multiple attempts to arrive at the correct solution, see 
subsection~\ref{subsec:system}). If all submissions were correct, meaning, every student would have solved the problem
correctly on the first attempt, the difficulty index would be 0. If none of the submissions were correct, the index would be 10.


\subsection{\label{subsec:disccat}Discussion Classification}
Student discussion entries were classified into three types and four features. The four types are
\begin{description}
\item[Emotional] - discussion contributions were classified as ``emotional" if they mostly communicated opinions,
complaints, gratitude, feelings, etc. Two subtypes were ``positive" and ``negative."
\item[Surface] - discussion contributions were classified as ``surface" if they dealt with surface features of the 
problem or where surface level requests for help. Two subtypes were ``question" and ``answer."
\item[Procedural] - contributions that describe or inquire about a mechanisms to solve the problem without
mention of the underlying concepts or reasoning. Two subtypes were ``question" and ``answer."
\item[Conceptual] - contributions that deal with the underlying concepts of the problem. Two subtypes were
``question" and ``answer."
\end{description}
In addition, discussion contributions were classified by the following features:
\begin{description}
\item[Unrelated] - the contribution is not related to the problem.
\item[Solution-oriented] - the goal of the contribution is to arrive at the correct answer without mentioning or
dealing with the mathematics or physics of the problem.
\item[Mathematical] - the contribution deals mostly with the mathematical aspects of the problem.
\item[Physics] - the contribution deals mostly with the physics aspects of the problem.
\end{description}
Table~\ref{table:examples} shows examples of contributions and their classification.
\begin{table*}
\caption{Examples of discussion contribution types and features.\label{table:examples}} 
\begin{ruledtabular}
\begin{tabular}{l|p{3.9cm}|p{3.9cm}|p{3.9cm}|p{3.9cm}}
&Unrelated&Solution&Math&Physics\\\hline
Emotional&
Why is it that homeworks are getting longer and longer?
&
Everyone keeps saying they figured it out, but no one is telling how. Please let us know because we have tried everything!
&
Bless your heart, and thank you for having the patience to explain this vector addition stuff to people like me who're really struggling with this vector and sin, cos stuff. It's
starting to all come together.
&
Sometimes, when I think of the word ``physics,'' I get a sickening feeling in the pit of my stomach. It's sort of like a burning sensation.
\\\hline
Surface&
If this is extra credit, does that mean it won't be on the exam?
&
Post the answers you know are correct for sure ... all do this .. and we'll get it.
&
What's an arctan?
&
``e'' for this equation is equal to one because it is a black body ... hope this helps.\\\hline
Procedural&

&
Use this formula: T(final) = (m1c1T1+m2c2T2) / (m1c1+m2c2). Convert temp to Kelvin and then for your final answer convert back to Cel.&
Thanks, I just realized it. I was supposed to solve for cos(c) by moving everything to the other side of the equation then take the cos$\hat{ }$-1 of that.&
Use equation for torque:\newline
torque = current * area * sin(90)\newline
It is 90 because it is a rectangle.\newline
Once you solve for torque multiply it by the N they give you and that is your answer. Make sure to convert your mA to A and cm to m before putting into equation.
\\\hline
Conceptual&
&
I thought you could use the equations for rolling without slipping ... can anyone clarify as to why not?
&
Do not add 90 degrees. Your answer depends on which quadrant your angle is in. You want the answer to be in the upper right quadrant, so add 180 to the absolute value of your answer if you have a negative x component value to find the angle you are looking for.
&
I have the correct answer, but I don't understand why it is correct. Why would there be an acceleration at the ball's highest point? Why wouldn't it be zero?
\end{tabular}
\end{ruledtabular}
\end{table*}
Discussion contributions were always classified as a whole, and since they were fairly short, they mostly fell clearly into one of the classes. If a longer contribution had aspects of more than one class, it was characterized by
the class that its majority fell into. Discussion contributions by teaching assistants and instructors were not 
considered. Also, the correctness of the posting was not considered, e.g., a discussion entry was considered ``conceptual'' even if it drew the wrong conclusions. 
 Table~\ref{table:disccat} shows the distribution of the available discussion contributions.
\begin{table}
\caption{Classification of the online discussion contributions according the classification scheme described in 
subsection~\ref{subsec:disccat}. The columns denote the different discussion types and subtypes, while the 
rows denote the 
features.\label{table:disccat}}
\begin{ruledtabular}
\begin{tabular}{lcccccccc|l}
&\multicolumn{2}{c}{Emotional}
&\multicolumn{2}{c}{Surface}
&\multicolumn{2}{c}{Procedural}
&\multicolumn{2}{c}{Conceptual}&\\

     &Pos&Neg&Q&A&Q&A&Q&A&\\
 Unrelated&   71&   54&   10&    1&     &     &    1&     &137\\
 Solution &  279&  185&  601&  341&  353&  456&   12&    3&2230\\
 Math     &    1&    6&   49&   36&   73&   87&    3&    6&261\\
 Physics  &     &   14&   85&   81&  170&  190&  100&  126&766\\\hline
          &  351&  259&  745&  459&  596&  733&  116&  135&3394
\end{tabular}
\end{ruledtabular}
\end{table}


In addition, the following 
superclasses are considered:
\begin{description}
\item[Chat] - all contributions that are unrelated or emotional.
\item[Emotional climate] - the number of positive non-unrelated contributions minus the number of negative non-unrelated
contributions. This number would be negative if the problem led to mostly negative emotional comments.
\item[Type and feature sums] - number of all related contributions belonging to a certain type, subtype, or feature.
\end{description}  

The majority of the discussion contributions were of type surface-level or procedural, followed by emotional 
contributions.
The vast majority of discussion contributions had the feature of being solution-oriented, 
yet a considerable number dealt with the physics
of the problems. 


\section{Results of Analysis by Student}
\subsection{Participation}
\begin{figure*}
\includegraphics[width=160mm]{KortemeyerFig4}% Here is how to import EPS art
\caption{\label{fig:contribBinned}Number of students versus number of discussion contributions.}
\end{figure*}
Within the first semester calculus-based course, an analysis by student was performed. Out of the 211 students in the course,
138 students (65 percent) contributed at least one discussion posting over the course of the semester. Figure~\ref{fig:contribBinned} shows the distribution
of number of discussion contributions over the course of the semester. Most students who participated made between one and ten contributions, but one student made
66 postings.
It is not possible to find out which percentage students {\it read} the discussions, since discussion are automatically attached to the questions and always visible.
The average number of postings per student was $5\pm0.7$. Women had a significantly higher average number of postings than men:
each female student contributed an average of $5.9\pm1$ postings, while each male student contributed an average of $3.7\pm0.7$ postings.
\subsection{Grade-Dependence of Discussion Contributions\label{subsec:gradedep}} 
The average grade in the course was $3.21\pm0.05$, with men and women achieving equally high grades (men: $3.29\pm0.08$; women: $3.17\pm0.05$). 
No correlation could be found between the average number of discussion postings and the grade in the course --- in terms of absolute 
numbers, within statistical errors, students with high and low grades in the course participated equally in the discussions. The positive correlation between participation in the
this ``moderated'' discussion forum and course grades~\cite{kashy03} could not be confirmed in this study.
\begin{figure}
\includegraphics[width=86mm]{KortemeyerFig5}% Here is how to import EPS art
\caption{\label{fig:gradecorrel}Prominance of discussion superclasses by grade.}
\end{figure}
Significant differences as a function of course grade appear when considering the classes of discussions (subsection~\ref{subsec:disccat}). 
In this analysis, the percentage prominance of certain types and 
features in students' cummulative contributions over the semester was analyzed. The individual percentage (relative) prominances were then averaged by grade. 
Note that the outcome is independent of the absolute number of postings a student made, e.g., the discussion behaviour of the student who made 66 contributions is weighed 
equally to that of a student having made only the average 5 contributions. Figure~\ref{fig:gradecorrel}
shows the outcome of this study by discussion superclass. As an example, the figure is to be interpreted this way: within the indicated errors, 
55 percent of a 3.0 student's discussion contributions were solution-oriented. The lines represent second-order polynomial fits to the data.

The relative prominance of solution-oriented discussion contributions varies most strongly with grade, from 75 percent for a 2.0 student to 45 percent for a 4.0 student.
The relative prominance of physics-related and conceptual discussion contributions on the other hand increases with grade.
The relative prominance of procedural discussions does not vary significantly with grades and is consistent with 42 percent promimance across grades and gender, except for the 23 female 4.0 students, where it is $68\pm7$ percent --- the 22 male 4.0 students, by comparison, average $34\pm8$ percent procedural discussions.

Except for the exceptionally high prominance of procedural discussion among the best female students, the results are not surprising, but verify the validity
of the classification approach.

At the same time, the results confirm that conceptual and physics-related discussions are positively correlated with success in the course, while solution-oriented discussion contributions are strongly negatively correlated. While cause and effect may be arguable, in the following 
section~\ref{sec:question}, particular attention needs to be paid to question properties that elicit either the desirable or undesirable discussion behavioral patterns.
 
\section{Results of Analysis by Question\label{sec:question}}
\subsection{Influence of Question Difficulty}
Each discussion contribution associated with a question was classified according to the types and features described in 
subsection~\ref{subsec:disccat}. As a measure of the prominence of a class in a given discussion, 
the number of contributions belonging to it is divided by the total number of contributions. The discussion characteristics of the problems were binned by their 
difficulty index and the average percentage plotted in figure~\ref{fig:diff}. Only superclasses are
shown (subsection~\ref{subsec:problemcat}), namely the emotional climate (crosses), as well as all (questions and answers) related
procedural 
(triangles) and conceptual (diamonds) contributions. As an example, the plot is to be interpreted in the following way: within the given
error boundaries, for a question with difficulty index of six, ten percent of the online discussion is conceptual.
\begin{figure}
\includegraphics[width=92mm]{KortemeyerFig6}% Here is how to import EPS art
\caption{\label{fig:diff}Discussion characteristics as a function of problem difficulty.
}
\end{figure}
In addition, the data was fit using second order (procedural, long dashes) and third order (emotional climate, short dashes; conceptual, solid) polynomials.

The greatest variation is found in the emotional climate of the discussion. As is to be expected, the climate is mostly positive
for ``easy" questions, but then remains positive for a fairly wide range of problem difficulties until it becomes negative
at a difficulty index of 7. Only six questions had a difficulty index of 9, and --- surprisingly --- none of these had
associated emotional comments.

For difficulty indizes beyond 3, the prominence of conceptual discussions increases. Surprisingly, it also increases for easier
questions. This may be attributed to students feeling more confident discussing easier problems on a conceptual level, or simply
in there being less of a need of procedural discussions.
Overall, the prominence of conceptual discussions is disappointingly low, as it varies between 5 and 16 percent.

Beyond a difficulty index of 5, within error boundaries, the prominence of conceptual discussions would be consistent with a constant 10 percent. If fostering them is a goal, 
and the emotional climate an indicator of ``pain,'' then beyond a difficulty index of 5 a significant increase in ``pain'' results in a non-significant gain. 

Across all difficulties, procedural contributions dominate the discussions, with relatively little significant variance around
the 40 percent mark. The maximum occurs for questions with a difficulty index of 5. 

In figure~\ref{fig:diffnochat} the same analysis was carried out, but this time excluding all ``chat" contributions 
(subsection~\ref{subsec:problemcat}), i.e., only related non-emotional contributions were considered. The relative prominence of procedural and conceptual discussions systematically 
increases, but all observations from the full analysis remain valid. ``Chat'' mostly provides a constant background across all difficulty indices. 
\begin{figure}
\includegraphics[width=92mm]{KortemeyerFig7}% Here is how to import EPS art
\caption{\label{fig:diffnochat}Discussion characteristics as a function of problem difficulty, no considering ``chat."
}
\end{figure}

\subsection{\label{subsec:qtype}Influence of Question Types}
Each question was classified according to the types and features described in subsection~\ref{subsec:problemcat}, and each associated discussion entry according to~\ref{subsec:disccat}. As a measure of the prominence of a class in a given discussion, 
the number of contributions belonging to it is divided by the total number of contributions. 
Table~\ref{table:qtype} shows the percentage prominence of discussion contributions with a certain type or with certain features in the discussions associated with questions
that are of a certain type or have certain features. 
\begin{table*}
\caption{Influence of question types and features on discussions.
The values indicate the percentage prominence of the discussion superclasses, types, and features (columns) for discussions associated with questions of a certain 
type or with certain features (rows). The values in brackets result from an analysis with ``chat'' excluded.\label{table:qtype}} 
\begin{ruledtabular}
\begin{tabular}{lcccccc}
&Emot. Clim.&Procedural&Solution&Math&Physics&Conceptual\\
Multiple Choice&-5$\pm$3&28$\pm$7 (29$\pm$8)&66$\pm$7 (74$\pm$7)&9$\pm$6 (9$\pm$6)&16$\pm$5 (17$\pm$5)&6$\pm$3 (7$\pm$3)\\
Short Textual&&&&&&\\
Numerical&4$\pm$1&48$\pm$1 (57$\pm$1)&52$\pm$1 (63$\pm$2)&8$\pm$1 (9$\pm$1)&23$\pm$1 (27$\pm$1)&7$\pm$1 (8$\pm$1)\\
Formula&6$\pm$8&29$\pm$11 (31$\pm$10)&57$\pm$16 (64$\pm$18)&31$\pm$16 (36$\pm$18)&&\\
Mult.-choice Mult.-resp.&1$\pm$1&15$\pm$3 (16$\pm$3)&66$\pm$4 (72$\pm$4)&1$\pm$1 (2$\pm$2)&22$\pm$3 (26$\pm$3)&14$\pm$2 (18$\pm$3)\\
Ranking&2$\pm$3&24$\pm$11 (26$\pm$12)&41$\pm$18 (46$\pm$20)&&52$\pm$20 (54$\pm$20)&38$\pm$18 (39$\pm$17)\\
Click-on-Image&0$\pm$9&14$\pm$6 (18$\pm$8)&53$\pm$8 (69$\pm$11)&3$\pm$3 (5$\pm$5)&25$\pm$11 (26$\pm$11)&22$\pm$8 (25$\pm$9)\\\hline
 
``Conventional''&4$\pm$1&42$\pm$1 (50$\pm$2)&55$\pm$1 (65$\pm$2)&7$\pm$1 (8$\pm$1)&23$\pm$1 (27$\pm$1)&9$\pm$1 (10$\pm$1)\\
Rep-Trans&-2$\pm$2&37$\pm$4 (45$\pm$4)&52$\pm$3 (63$\pm$4)&7$\pm$2 (9$\pm$2)&23$\pm$3 (28$\pm$3)&8$\pm$2 (10$\pm$2)\\
\end{tabular}
\end{ruledtabular}
\end{table*}
Error boundaries on the emotional climate values are rather large and mostly include zero (neutral), indicating no significant preferences within the limited sample.
Yet, students clearly dislike multiple-choice questions, while they clearly like numerical answer problems. The data also indicates that students prefer ``conventional'' over
representation-translation problems.

The prominence of procedural discussions is significantly higher for numerical problems than for any other problem types, and higher for ``conventional'' than for
representation-translation problems. The latter difference vanishes when ``chat'' is excluded.

Solution-oriented contributions are significantly higher for multiple-choice and multiple-choice-multiple-response problems than for the other problem types with the exception 
of formula-response questions, where error-boundaries overlap. In spite of the randomization provided, in discussion entries, students frequently reverse-engineered the complete randomization space by copying their correct answer screens into the discussions 
(see the example for a surface-level solution-oriented discussion entry in Table~\ref{table:examples}). 

The prominence of mathematical discussion contributions is the highest for formula-response questions, approximately equal for numerical and single-response multiple-choice questions, and the lowest for multiple-choice-multiple-response, ranking, and click-on-image questions.

The prominence of physics-related discussion contributions was the highest for ranking and click-on-image problems, and the lowest for multiple-choice questions.

Finally, when it comes to conceptual discussions, their prominence is significantly lower in single-response multiple-choice and numerical problems than in the other problem types. In the 
earlier study by Kashy~\cite{kashyd01}, it was also found that mastery of these same question types does not predict overall performance on the final exam as well as other question types. 
Multiple-choice problems that do not involve numbers are frequently called ``conceptual'' questions, but in this study, it was found that they do not necessarily lead to conceptual discussions.

It is a surprising result that the only significant difference between ``conventional'' and representation-translation problems is that students discuss slightly less procedure in favor of 
more complaints, and that differences disappear when ``chat'' is excluded from the analysis. McDermott~\cite{mcdermott} and Beichner~\cite{beichner} on the other hand found that students have unexpected difficulties in translating for example data presented as graphs, so a stronger effect of this feature was expected. In additon, Kashy~\cite{kashyd01} found that mastery of representation-translation problems 
is the best predictor of final exam scores, even when controlling for ACT, cumulative GPA, and force-concept inventory pretests.
Discussion behavior and final exam performance are clearly different measurements for the influence of problem types and do not necessarily need to correlate, but a connection between 
individual discussion behavior and performance in the course clearly exists (see subsection~\ref{subsec:gradedep}).
It should be noted that the earlier study dealt with a relatively small set of
representation-translation problems, some of which involved non-static time-evolving simulations as data-source, while in this study, none of the simulation-based problems were assigned. A future study may need to consider the interpretation of time-evolving 
simulations as a separate feature, once that more problems of this type exist in the resource pool.
\subsection{Influence of course}
Few significant differences could be found between the algebra-based and the calculus-based course:
\begin{itemize}
\item discussions in the algebra-based course had a significantly higher emotional
climate (6$\pm$1 versus 2$\pm$1)
\item the algebra-based course had a higher prominence of ``chat'' (21$\pm$2\% versus 11$\pm$1\% (first semester) and 14$\pm$2\% (second semester))
\item physics-related discussion were significantly higher in the calculus-based course (28$\pm$2\% (first semester) and 23$\pm$2\% (second semester)) versus 17$\pm$2\% in the algebra-based course.
\item conceptual-discussions were significantly higher in the first semester of the calculus-based course (12$\pm$2\% (calculus, first semester) versus 6$\pm$2\% (algebra)), but this difference vanished in the second semester (7$\pm$1\% (calculus, second semester)).
\end{itemize}  
Especially the last observation is discouraging, since as the students in the calculus-based course progressed further into their study of physics, the degree to which they were discussing concepts
decreased. This might partly be due to the different subject matter (electricity and magnetism versus mechanics), but also due to the lack of reward for conceptual considerations in solving standard
homework problems~\cite{lin}. 
\subsection{Qualitative Observations}
Reading the online discussions associated with the homework provides valuable insights to the instructor, which are hard to quantify.
When assigning homework, instructors usually have an instructional goal in mind, for example, they would like the students to grapple with a certain concept or work through a specific strategy of problem 
solving. Until the ``reality check,'' the fact that a specific problem only serves this purpose when being approached with an expert mindset is under-appreciated. An even deeper misconception is the
assumption that solving the problem correctly is a reliable indicator of the concept or problem solving strategy being successfully communicated. What the (expert) instructor had in mind, and what the
(novice) learner actually does, can be worlds apart~\cite{lin,chi}. Students are going through reasoning processes and steps that are hardly imaginable to the instructor, and more often than not do several times more work
than necessary. The situation that they get a problem right for the wrong reasons is rare, but the instances that they get the problem correct with the same (minimal) amount of steps that an expert 
would are equally rare --- in the end, the concept that was meant to be communicated is lost.

Many of these shortcomings may be correctable through early detection, and closely following the online student discussions prior to lecture, particularly around the assigned reading problems, may be a valid extension of the Just-in-Time Teaching~\cite{jitt} technique.
\section{Conclusions}
Online student discussions are a rich source of insight into student problem solving behavior. It was verified that indeed conceptual and physics-related discussion contributions are characteristics of students who are successful in the course, while the prominance of solution-oriented
discussion contributions is strongly negatively correlated with success in the course.

Different discussion patterns ensue around different question characteristics:
\begin{description}
\item[Difficulty] Very easy problems can elicit a high level conceptual discussions,
and so can problems of mid-range difficulty. As problems become more difficult, there is no significant gain in conceptual discussions.
\item[Question Types] Different question types result in different association discussion patterns. Discussions on a procedural level are more prominent for numerical problems than for any other problem type. Solution-oriented discussions are more prominent for multiple-choice style questions in an effort to short-circuit the conceptual reasoning.
Discussions around single-response multiple choice questions and numerical questions have a significantly lower prominance of conceptual discussions than other question types.
Ranking questions show very favorable discussion patterns, but their sample size has been too small to make definitive statements. 
\end{description}
Analyzing online discussions around problems has been found to provide valuable insights into student problem-solving strategies. 
\begin{acknowledgments}
Supported by the National Science Foundation under NSF-ITR 0085921 and NSF-CCLI-ASA 0243126. Any opinions, findings, and conclusions or recommendations expressed in this 
publication are those of the author and do not necessarily reflect the views of the National Science Foundation.
\end{acknowledgments}

\bibliography{discussions}% Produces the bibliography via BibTeX.

\end{document}

--www1134268129--