[LON-CAPA-cvs] cvs: modules /gerd/discussions/paper discussions.tex
www
lon-capa-cvs@mail.lon-capa.org
Tue, 05 Apr 2005 11:28:57 -0000
This is a MIME encoded message
--www1112700537
Content-Type: text/plain
www Tue Apr 5 07:28:57 2005 EDT
Modified files:
/modules/gerd/discussions/paper discussions.tex
Log:
More writing
--www1112700537
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20050405072857.txt"
Index: modules/gerd/discussions/paper/discussions.tex
diff -u modules/gerd/discussions/paper/discussions.tex:1.7 modules/gerd/discussions/paper/discussions.tex:1.8
--- modules/gerd/discussions/paper/discussions.tex:1.7 Mon Apr 4 11:56:22 2005
+++ modules/gerd/discussions/paper/discussions.tex Tue Apr 5 07:28:56 2005
@@ -57,8 +57,10 @@
LON-CAPA allows for immediate feedback on problem correctness to the student, as well as multiple tries to arrive at
the correct solution (both features could be disabled by the instructor). LON-CAPA is designed to foster
communication among the learners, and asynchronous threaded discussion boards
-are attached directly to the bottom of every online resource. For the purposes of this project, it is thus possible
+are attached directly to the bottom of every online resource.
+For the purposes of this project, it is thus possible
to establish a one-to-one association between an online problems and discussions.
+Students can post anonymously or using a screenname, however, the full name is always visible to the instructor.
In addition, LON-CAPA keeps statistical data for every problem, which allows to associate problems with their
degree of difficulty.
@@ -161,11 +163,57 @@
\item[Mathematical] - the contribution deals mostly with the mathematical aspects of the problem.
\item[Physics] - the contribution deals mostly with the physics aspects of the problem.
\end{description}
-The following are examples of how contributions were characterized:
-
+Table~\ref{table:examples} shows examples of contributions and their classification.
+\begin{table*}
+\caption{Influence of question types and features on discussions.
+The values indicate the percentage prominence of the discussion superclasses, types, and features (columns) for discussions associated with questions of a certain
+type or with certain features (rows). The values in brackets result from an analysis with ``chat'' excluded.\label{table:examples}}
+\begin{ruledtabular}
+\begin{tabular}{l|p{3.9cm}|p{3.9cm}|p{3.9cm}|p{3.9cm}}
+&Unrelated&Solution&Math&Physics\\\hline
+Emotional&
+Why is it that homeworks are getting longer and longer?
+&
+Everyone keeps saying they figured it out, but no one is telling how. Please let us know because we have tried everything!
+&&\\\hline
+Surface&
+If this is extra credit, does that mean it won't be on the exam?
+&
+Post the answers you know are correct for sure ... all do this .. and we'll get it.
+&
+What's an arctan?
+&
+``e'' for this equation is equal to one because it is a black body ... hope this helps.\\\hline
+Procedural&
+
+&
+Use this formula: T(final) = (m1c1T1+m2c2T2) / (m1c1+m2c2). Convert temp to Kelvin and then for your final answer convert back to Cel.&
+Thanks, I just realized it. I was supposed to solve for cos(c) by moving everything to the other side of the equation then take the cos$\hat{ }$-1 of that.&
+Conservation of Energy Problem\newline
+KE(initial)+Heat Energy(initial)=Latent Heat\newline
+so ... .5mv$\hat{ }$2(bullet)+m*(c-lead)*(change in T)=m(of the water)*L\newline
+m(of the water) is what you want to find\newline
+L is L h2o fusion - given\newline
+m - mass of bullet - given\newline
+v- given\newline
+You have to convert the c(lead) and the L h20 fusion, you do this by multiplying them by (4.186)(1000)\newline
+The final answer units are Kg.\newline
+I don't know if that makes sense .. but I hope so.\\\hline
+Conceptual&
+&
+I thought you could use the equations for rolling without slipping ... can anyone clarify as to why not?
+&
+Why can't you call Vf negative, and say Vi is moving in the positive direction? Depending on what you call positive and negative, you get two different answers for Vf (two different numbers, not just
+a different sign).
+&
+I have the correct answer, but I don't understand why it is correct. Why would there be an acceleration at the ball's highest point? Why wouldn't it be zero?
+\end{tabular}
+\end{ruledtabular}
+\end{table*}
Discussion contributions were always classified as a whole, and since they were fairly short, they mostly fell clearly into one of the classes. If a longer contribution had aspects of more than one class, it was characterized by
the class that its majority fell into. Discussion contributions by teaching assistants and instructors were not
-considered. Table~\ref{table:disccat} shows the distribution of the available discussion contributions.
+considered. Also, the correctness of the posting was not considered, e.g., a discussion entry was considered ``conceptual'' even if it drew the wrong conclusions.
+ Table~\ref{table:disccat} shows the distribution of the available discussion contributions.
\begin{table}
\caption{Classification of the online discussion contributions according the classification scheme described in
subsection~\ref{subsec:disccat}. The columns denote the different discussion types and subtypes, while the
@@ -195,27 +243,27 @@
\subsection{Computation of Results\label{subsec:comp}}
Each question was classified according to the types and features described in subsection~\ref{subsec:problemcat}. In addition, its
difficulty index was computed according to the formula
-\begin{equation}\label{eqn:diffidx}
-\mbox{Difficulty Index}=10\left(1-\frac{N_{\mbox{correct}}{N_{\mbox{attempts}}\right)
-\end{equation}
+\begin{equation*}\label{eqn:diffidx}
+\mbox{Difficulty Index}=10\left(1-\frac{N_{\mbox{correct}}}{N_{\mbox{attempts}}}\right)
+\end{equation*}
where $N_{\mbox{correct}}$ is the total number of correct solution in the course, and $N_{\mbox{attempts}}$ is the total number of
correct and incorrect solution submissions (the system allows multiple attempts to arrive at the correct solution, see
subsection~\ref{subsec:system}). If all submissions were correct, meaning, every student would have solved the problem
correctly on the first attempt, the difficulty index would be 0. If none of the submissions were correct, the index would be 10.
Each associated discussion contribution was classified according to the types and features described in
-subsection~\ref{subsec:disccat}. As a measure of the prominance of a class in a given discussion,
-the number of contributions belonging to it is devided by the total number of contributions. In addition, the following
+subsection~\ref{subsec:disccat}. As a measure of the prominence of a class in a given discussion,
+the number of contributions belonging to it is divided by the total number of contributions. In addition, the following
superclasses are considered:
\begin{description}
\item[Chat] - all contributions that are unrelated or emotional.
\item[Emotional climate] - the number of positive non-unrelated contributions minus the number of negative non-unrelated
contributions. This number would be negative if the problem led to mostly negative emotional comments.
-\irem[Type and feature sums] - number of all related contributions belonging to a certain type, subtype, or feature.
+\item[Type and feature sums] - number of all related contributions belonging to a certain type, subtype, or feature.
\end{description}
\section{Results}
\subsection{Influence of Question Difficulty}
-The discussion characteristics of the problems were bined by their
+The discussion characteristics of the problems were binned by their
difficulty index (equation~\ref{eqn:diffidx}) and the average percentage plotted in figure~\ref{fig:diff}. Only superclasses are
shown (subsection~\ref{subsec:comp}), namely the emotional climate (crosses), as well as all (questions and answers) related
procedural
@@ -233,20 +281,20 @@
at a difficulty index of 7. Only six questions had a difficulty index of 9, and --- surprisingly --- none of these had
associated emotional comments.
-For difficulty indizes beyond 3, the prominance of conceptual discussions increases. Surprisingly, it also increases for easier
+For difficulty indizes beyond 3, the prominence of conceptual discussions increases. Surprisingly, it also increases for easier
questions. This may be attributed to students feeling more confident discussing easier problems on a conceptual level, or simply
in there being less of a need of procedural discussions.
-Overall, the prominance of conceptual discussions is disappointingly low, as it varies between 5 and 16 percent.
+Overall, the prominence of conceptual discussions is disappointingly low, as it varies between 5 and 16 percent.
-Beyond a difficulty index of 5, within error boundaries, the prominance of conceptual discussions would be consistant with a constant 10 percent. If fostering them is a goal,
+Beyond a difficulty index of 5, within error boundaries, the prominence of conceptual discussions would be consistent with a constant 10 percent. If fostering them is a goal,
and the emotional climate an indicator of ``pain,'' then beyond a difficulty index of 5 a significant increase in ``pain'' results in a non-significant gain.
Across all difficulties, procedural contributions dominate the discussions, with relatively little significant variance around
the 40 percent mark. The maximum occurs for questions with a difficulty index of 5.
In figure~\ref{fig:diffnochat} the same analysis was carried out, but this time excluding all ``chat" contributions
-(subsection~\ref{subsec:comp}), i.e., only related non-emotional contributions were considered. The relative prominance of procedural and conceptual discussions systematically
-increaeses, but all observations from the full analysis remain valid. ``Chat'' mostly provides a constant background across all difficulty indices.
+(subsection~\ref{subsec:comp}), i.e., only related non-emotional contributions were considered. The relative prominence of procedural and conceptual discussions systematically
+increases, but all observations from the full analysis remain valid. ``Chat'' mostly provides a constant background across all difficulty indices.
\begin{figure}
\includegraphics[width=92mm]{diffnochat}% Here is how to import EPS art
\caption{\label{fig:diffnochat}Discussion characteristics as a function of problem difficulty, no considering ``chat."
@@ -254,11 +302,11 @@
\end{figure}
\subsection{\label{subsec:qtype}Influence of Question Types}
-Table~\ref{table:qtype} shows the percentage prominance of discussion contributions with a certain type or with certain features in the discussions associated with questions
+Table~\ref{table:qtype} shows the percentage prominence of discussion contributions with a certain type or with certain features in the discussions associated with questions
that are of a certain type or have certain features.
\begin{table*}
\caption{Influence of question types and features on discussions.
-The values indicate the percentage prominance of the discussion supertypes, types, and features (columns) for discussions associated with questions of a certain
+The values indicate the percentage prominence of the discussion superclasses, types, and features (columns) for discussions associated with questions of a certain
type or with certain features (rows). The values in brackets result from an analysis with ``chat'' excluded.\label{table:qtype}}
\begin{ruledtabular}
\begin{tabular}{lcccccc}
@@ -280,30 +328,41 @@
Yet, students clearly dislike multiple-choice questions, while they clearly like numerical answer problems. The data also indicates that students prefer ``conventional'' over
representation-translation problems.
-The prominance of procedural discussions is significantly higher for numerical problems than for any other problem types, and higher for ``conventional'' than for
+The prominence of procedural discussions is significantly higher for numerical problems than for any other problem types, and higher for ``conventional'' than for
representation-translation problems. The latter difference vanishes when ``chat'' is excluded.
Solution-oriented contributions are significantly higher for multiple-choice and multiple-choice-multiple-response problems than for the other problem types with the exception
of formula-response questions, where error-boundaries overlap.
-The prominance of mathematical discussion contributions is the highest for formula-response questions, approximately equal for numerical and multiple-choice questions, and the lowest for multiple-choice-multiple-response, ranking, and click-on-image questions.
+The prominence of mathematical discussion contributions is the highest for formula-response questions, approximately equal for numerical and single-response multiple-choice questions, and the lowest for multiple-choice-multiple-response, ranking, and click-on-image questions.
-The prominance of physics-related discussion contributions was the highest for ranking and click-on-image problems, and the lowest for multiple-choice questions.
+The prominence of physics-related discussion contributions was the highest for ranking and click-on-image problems, and the lowest for multiple-choice questions.
-Finally, when it comes to conceptual discussions, their prominance is signifcantly lower in multiple-choice and numerical problems than in the other problem types.
+Finally, when it comes to conceptual discussions, their prominence is significantly lower in single-response multiple-choice and numerical problems than in the other problem types.
It is a surprising result that the only significant difference between ``conventional'' and representation-translation problems is discuss slightly less procedure in favor of
more complaints, and that differences disappear when ``chat'' is excluded from the analysis.
\subsection{Influence of course}
-Few signficant differences could be found between the algebra-based and the calculus-based course:
+Few significant differences could be found between the algebra-based and the calculus-based course:
\begin{itemize}
\item discussions in the algebra-based course had a significantly higher emotional
climate (6$\pm$1 versus 2$\pm$1)
-\item the algebra-based course had a higher prominance of ``chat'' (21$\pm$2\% versus 11$\pm$1\% (first semester) and 14$\pm$2\% (second semester))
+\item the algebra-based course had a higher prominence of ``chat'' (21$\pm$2\% versus 11$\pm$1\% (first semester) and 14$\pm$2\% (second semester))
\item physics-related discussion were significantly higher in the calculus-based course (28$\pm$2\% (first semester) and 23$\pm$2\% (second semester) versus 17$\pm$2\%)
\item conceptual-discussions were significantly higher in the first semester of the calculus-based course (12$\pm$2\% versus 6$\pm$2\%), but this difference vanished in the second semester (7$\pm$1\%).
\end{itemize}
-
+Especially the last observation is discouraging, since as the student in the calculus-based course progressed further into their study of physics, the degree to which they were discussing concepts
+decreased. This might partly be due to the different subject matter (electricity and magnetism versus mechanics), but also due to the lack of reward for conceptual considerations in solving standard
+homework problems.
+\subsection{Qualitative Observations}
+Reading the online discussions associated with the homework provides valuable insights to the instructor, which are hard to quantify.
+When assigning homework, instructors usually have an instructional goal in mind, for example, they would like the students to grapple with a certain concept or work through a specific strategy of problem
+solving. Until the ``reality check,'' the fact that a specific problem only serves this purpose when being approached with an expert mindset is under-appreciated. An even deeper misconception is the
+assumption that solving the problem correctly is a reliable indicator of the concept or problem solving strategy being successfully communicated. What the (expert) instructor had in mind, and what the
+(novice) learner actually does, is worlds apart. Students are going through reasoning processes and steps that are hardly imaginable to the instructor, and more often than not do several times more work
+than necessary. The situation that they get a problem right for the wrong reasons is rare, but the instances that they get the problem correct with the same (minimal) amount of steps that an expert
+would are equally rare --- in the end, the concept that was meant to be communicated is lost.
+\section{Conclusions}
\begin{acknowledgments}
Supported by the National Science Foundation under NSF-ITR 0085921 and NSF-CCLI-ASA 0243126. Any opinions, findings, and conclusions or recommendations expressed in this
--www1112700537--