[LON-CAPA-cvs] cvs: modules /gerd/correlpaper correlations.tex
www
lon-capa-cvs@mail.lon-capa.org
Sat, 30 Sep 2006 23:45:48 -0000
This is a MIME encoded message
--www1159659948
Content-Type: text/plain
www Sat Sep 30 19:45:48 2006 EDT
Modified files:
/modules/gerd/correlpaper correlations.tex
Log:
Corrections.
--www1159659948
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20060930194548.txt"
Index: modules/gerd/correlpaper/correlations.tex
diff -u modules/gerd/correlpaper/correlations.tex:1.14 modules/gerd/correlpaper/correlations.tex:1.15
--- modules/gerd/correlpaper/correlations.tex:1.14 Sat Sep 30 16:35:10 2006
+++ modules/gerd/correlpaper/correlations.tex Sat Sep 30 19:45:46 2006
@@ -51,11 +51,11 @@
% but any date may be explicitly specified
\begin{abstract}
-An important result of Physics Education Research is that students' learning and success in a course is correlated with their beliefs, attitudes, and expectations regarding physics. However, it is hard to assess these beliefs, and traditional survey instruments such as the Maryland Physics Expectations Survey (MPEX) are intended to evaluate the impact of one or more semesters of instruction on an overall class and improve teaching.
+An important result of Physics Education Research is that students' learning and success in a course is correlated with their beliefs, attitudes, and expectations regarding physics. However, it is hard to assess these beliefs for individual students, and traditional survey instruments such as the Maryland Physics Expectations Survey (MPEX) are intended to evaluate the impact of one or more semesters of instruction on an overall class and improve teaching.
In this study, we investigate the possibility of using the analysis of online student discussion behavior as an indicator of an individual student's approach to physics. These discussions are not tainted by the effects of self-reporting, and are gathered in authentic non-research settings, where students attempt to solve problems in the way that they belief is most efficient and appropriate.
-We investigate correlations with a traditional instrument, namely the MPEX, as well as correlations with the Force Concept Inventory (FCI), the final exam grade, and the overall course performance as a measure of the student's learning. To gauge the outcomes, we also investigate correlations between these measures.\end{abstract}
+We calculate the correlation of both MPEX and student discussions with different measures of student learning, and find that on an individual base, student discussions are a stronger predictor of success than MPEX outcomes.\end{abstract}
\pacs{01.40.Fk}% PACS, the Physics and Astronomy
% Classification Scheme.
@@ -67,7 +67,7 @@
The MPEX makes the limitations of this approach very explicit in their ``Product Warning Label''~\cite{mpexwarning}: ``students often think that they function in one fashion and actually behave differently. For the diagnosis of the difficulties of individual students more detailed observation is required.'' Online student discussions associated with online physics problems are different in that they are generated within the real context of the course, and students have a vested interest in making these discussions as productive as possible, given their understanding of how physics is done and their approach to it. They could thus be a ``reality check'' of students' beliefs, attitudes, and expectations.
-The MPEX ``Product Warning Label'' continues, that ``this survey is primarily intended to evaluate the impact of one or more semesters of instruction on an overall class''~\cite{mpexwarning}, and recommends using the outcomes, in combination with evaluations of student learning of content, as a means to improve overall course instruction. In this paper, we are asking the question if the evaluation of student online discussion behavior can be used as a means to get to assess the attitudes and beliefs of an individual student, and if in turn, these can be used to predict the success of an individual student in the learning of physics content. An obvious application would be the early detection of students at risk.
+The MPEX ``Product Warning Label'' continues, that ``this survey is primarily intended to evaluate the impact of one or more semesters of instruction on an overall class''~\cite{mpexwarning}, and recommends using the outcomes, in combination with evaluations of student learning of content, as a means to improve overall course instruction. In this paper, we are asking the question if the evaluation of student online discussion behavior can be used as a means to assess the attitudes and beliefs of an individual student, and if in turn, these can be used to predict the success of an individual student in the learning of physics content. An obvious application would be the early detection of students at risk.
In particular:
\begin{itemize}
@@ -79,7 +79,7 @@
\section{\label{background}Background}
Previous studies indicate that correlations between epistemological beliefs and academic performance exist, both directly and indirectly \cite{schommer93,may02}. The problem is how to measure these beliefs, and techniques include surveys, guided interviews, and observations.
-Many of these, though, take place in artificial research settings and outside the normal course activity over a relatively short time, and research results regarding their predictive power are not conclusive: Coletta and Philips~\cite{coletta05} found a strong correlation between the FCI Gain and the MPEX Score, while Dancy~\cite{dancy02} found low correlations between the MPEX and the the performance on homework, tests, and final exams. The discrepancies might all be traced back to the ``Product Warning Lab''~\cite{mpexwarning}, that the survey is best used to gain insights into the beliefs of the class as a whole, rather than on an individual level.
+Research results regarding their predictive power of these instruments is not always conclusive: for example, Coletta and Philips~\cite{coletta05} found a strong correlation between the MPEX and FCI Gain, while Dancy~\cite{dancy02} found low correlations between the MPEX and the the performance on homework, tests, and final exams. The discrepancies might all be traced back to the ``Product Warning Lab''~\cite{mpexwarning}, that the survey is best used to gain insights into the beliefs of the class as a whole, rather than on an individual level.
Online discussions take place within the regular course context and over its complete duration. They are a rich source of feedback to the instructor~\cite{kortemeyer05feedback}, and their quality and character was found to be correlated with the type and difficulty of the associated problems~\cite{kortemeyer05ana}, i.e., data exists regarding the influence of {\it problem} characteristics on associated discussions. Unfortunately, less data exists on the correlation between {\it student} characteristics and discussion behavior, because usually only very few student characteristics are known, with the exception of the students' overall performance in the course. Thus, one of the few findings was the fact that certain discussion behavior, most prominently exhibited on ``non-sanctioned'' discussion sites external to the course, is negatively correlated with performance in the course~\cite{kashy03,kortemeyer05ana}.
@@ -205,7 +205,8 @@
As already found in Ref.~\cite{kortemeyer05ana}, most students are quite prolific in their online discussions, but a few students only made a small number of contributions, leading to small statistics on their actual discussion behavior. For each of the discussion correlations, we thus also carried out a second calculation limited to students who contributed at least five entries over the course of the semester.
\subsection{\label{mpex}The MPEX}
-We deployed the Maryland Physics Expectations Survey (MPEX)\cite{mpex} both at the beginning and the end of the mechanics semester. Participation was voluntary. We calculated the score in comparison to the ``favorable" expert responses given in Ref.~\cite{mpex} on the final (post) deployment, as well as, for students who participated both times, the gain. The same analysis was done for each cluster of the MPEX (example statements are given, including the expert answer):
+We deployed the Maryland Physics Expectations Survey (MPEX)\cite{mpex} both at the beginning and the end of the mechanics semester. Participation was voluntary. We calculated the ``score'' in comparison to the ``favorable" expert responses given in Ref.~\cite{mpex} -- please note that the
+word ``score'' in the context of the MPEX is thus not an absolute measure of correctness, but of agreement with the majority of an expert group, who does not even necessarily agree among each other. We calculated the final (post) deployment score, as well as, for students who participated both times, the gain. The same analysis was done for each cluster of the MPEX (example statements are given, including the expert answer):
\begin{itemize}
\item{\it Independence:} student takes responsibility for constructing their own understanding, rather than takes what is given by authorities (teacher, materials) without evaluation
\begin{quote}
@@ -232,7 +233,7 @@
Favorable: I go over my class notes carefully to prepare for tests in this course.
\end{quote}
\end{itemize}
-The overall scores of the students on the MPEX clusters were low (Independence 42\%; Coherence 46\%; Concepts 48\%; Reality Link 55\%; Math Link 40\%; Effort 47\%).
+The overall scores (i.e., agreement with the expert group) of the students on the MPEX clusters were low (Independence 42\%; Coherence 46\%; Concepts 48\%; Reality Link 55\%; Math Link 40\%; Effort 47\%).
\subsection{\label{performance}Measures of Student Learning}
As a measure of student conceptual understanding and learning, we deployed the revised Force Concept Inventory (FCI)\cite{fci} at the beginning and the end of the course, again with voluntary participation. As an additional measure of student performance, the performance on the final exam and the course grade for each student were taken into consideration. For the grade we used the raw percentage score, not the number grades, since it provides finer grained information about the overall student performance in the course.
\section{\label{perception}Student Perception of the Online Discussions and Survey Instruments}
@@ -251,41 +252,41 @@
\section{\label{results}Correlation Results}
\subsection{\label{MPEXDiscussion}Correlations between Discussion Behavior and MPEX}
-To directly compare the attitudes and beliefs measures, we calculated correlations between the prominence of discussion behavior classes and the MPEX clusters, and generally found them to be very low. As an example, the correlation between the score on the Concepts Cluster and the prominence of conceptual discussion contributions turned out to be $R=0.14 [-0.08 \to 0.34]; n=84$ when considering all students, and $R=0.15 [-0.13 \to 0.41]; n=51$ when only considering those who made at least five discussion contributions -- the 95\% confidence intervals (given in square brackets) include zero. Thus, we conclude that discussion behavior and the individual MPEX cluster scores are -- if at all -- only weakly correlated.
+To directly compare the attitudes and beliefs measures, we calculated correlations between the prominence of discussion behavior classes and the MPEX clusters, and generally found them to be very low. As an example, the correlation between the score on the Concepts Cluster and the prominence of conceptual discussion contributions turned out to be $R=0.14\ [-0.08\to0.34] (n=84)$ when considering all students, and $R=0.15\ [-0.13\to0.41] (n=51)$ when only considering those who made at least five discussion contributions -- the 95\% confidence intervals (given in square brackets) include zero. Thus, we conclude that discussion behavior and the individual MPEX cluster scores are -- if at all -- only weakly correlated.
\subsection{\label{learningcorreldis}Correlations between Discussions and Learning}
Figure~\ref{physicsgrade} shows the correlation between the prominence of physics-related discussions and the course grade percentage (for better statistics, only students who contributed at least five discussion entries over the course of the semester were considered).
\begin{figure}
\includegraphics[width=9cm]{physicsgrade}
-\caption{\label{physicsgrade}Correlation of percentage physics-related discussions with grade percentage ($R=0.33 [0.15 \to 0.49]$; $n=111$).}
+\caption{\label{physicsgrade}Correlation of percentage physics-related discussions with grade percentage ($R=0.33\ [0.15\to0.49] (n=111)$).}
\end{figure}
-Figure~\ref{fciphysics} shows how the percentage of a particular student's discussion contribution that was classified as "physics-related" correlates with their final FCI score ($R=0.34 [0.15 \to 0.51]$; $n= 95$). As already in Fig.~\ref{physicsgrade}, an additional analysis was carried out that was limited to students for which better statistics were available, which let to a stronger correlation ($R=0.51[0.29 \to 0.68]$; $n=57$). While physics-related discussions positively correlate with FCI scores and grades (Fig.~\ref{physicsgrade}), solution-oriented discussions negatively correlate (Fig.~\ref{solutionfci}; $R=-0.58 [-0.73 \to -0.38]$; $n=57$).
+Figure~\ref{fciphysics} shows how the percentage of a particular student's discussion contribution that was classified as "physics-related" correlates with their final FCI score ($R=0.51\ [0.29\to0.68] (n=57)$). While physics-related discussions positively correlate with FCI scores and grades (Fig.~\ref{physicsgrade}), solution-oriented discussions negatively correlate (Fig.~\ref{solutionfci}; $R=-0.58\ [-0.73\to-0.38] (n=57)$).
-\begin{figure*}
-\includegraphics[width=9cm]{fcipostphysics}\includegraphics[width=9cm]{fcipostphysicsT}
-\caption{\label{fciphysics}Correlation between the FCI score and the percentage of that student's discussion that was classified as "physics" ($R=0.34 [0.15 \to 0.51]$; $n= 95$). The figure on the right only includes students who contributed more than five discussion entries over the course of the semester ($R=0.51 [0.29 \to 0.68]$; $n=57$).}
-\end{figure*}
+\begin{figure}
+\includegraphics[width=9cm]{fcipostphysicsT}
+\caption{\label{fciphysics}Correlation of percentage physics-related discussions with final FCI score ($R=0.51\ [0.29\to0.68] (n=57)$).}
+\end{figure}
\begin{figure}
\includegraphics[width=9cm]{fcipostsolutionT}
-\caption{\label{solutionfci}Correlation of percentage solution-oriented discussions with final FCI score ($R=-0.58 [-0.73 \to -0.38]$; $n=57$).}
+\caption{\label{solutionfci}Correlation of percentage solution-oriented discussions with final FCI score ($R=-0.58\ [-0.73\to-0.38] (n=57)$).}
\end{figure}
\subsection{\label{learningcorrelmpex}Correlations between MPEX and Learning}
-Correlations between the MPEX and measures of student learning are generally weak. Considering final exam, FCI, and course grade, $R=0.36 [0.17 \to 0.52]$ ($n=97$) between the score on the Coherence cluster and the course grade percentage is the highest correlation found.
+Correlations between the MPEX and measures of student learning are generally weak. Considering final exam, FCI, and course grade, $R=0.36\ [0.17\to0.52] (n=97)$ between the score on the Coherence cluster and the course grade percentage is the highest correlation found.
Dancy~\cite{dancy02} found similarly low correlations with the performance on homework, tests, and final exams: direct comparison with the performance on the final exams found $R=0.37$ for the correlation with the total MPEX score ($R=0.27$ here), $R=0.39$ with the Independence Cluster ($R=0.25$ here), $R=0.24$ with the Coherence Cluster ($R=0.36$ here), $R=0.29$ with the Concept Cluster ($R=0.25$ here), $R=-.02$ with the Reality Link cluster ($R=0.1$ here), $R=0.3$ with the Math Link cluster (no significant correlation found here), and no significant correlation with the Effort Cluster ($R=0.1$ here).
-Figure~\ref{mpexfci} shows how the final MPEX and FCI scores correlated with each other, i.e, $R=0.24 [0.04 \to 0.42]$ ($n=97$).
-Coletta and Philips~\cite{coletta05} found a strong correlation between the FCI Gain and the MPEX Score ($R=0.52 [0.24 \to 0.72]; n=37$), while the same correlation turned out much lower in this study ($R=0.17 [-0.05 \to 0.37]; n=84$ here). The correlations reported here are in the same range that
+Figure~\ref{mpexfci} shows how the final MPEX and FCI scores correlated with each other, i.e, $R=0.24\ [0.04\to0.42] (n=97)$.
+Coletta and Philips~\cite{coletta05} found a strong correlation between the FCI Gain and the MPEX Score ($R=0.52\ [0.24\to0.72] (n=37)$), while the same correlation turned out much lower in this study ($R=0.17\ [-0.05\to0.37] (n=84)$ here). The correlations reported here are in the same range that
Perkins et al.~\cite{perkins04} found when investigating the influence of beliefs on conceptual learning, using the CLASS~\cite{adams04} and the Force and Motion Conceptual Evaluation (FMCE)~\cite{thornton98} instruments.
\begin{figure}
\includegraphics[width=9cm]{fcipostmpexpost}
-\caption{\label{mpexfci}Correlation of the final FCI score with the MPEX score ($R=0.24 [0.04 \to 0.42]$; $n=97$).}
+\caption{\label{mpexfci}Correlation of the final FCI score with the MPEX score ($R=0.24 [0.04 \to 0.42] (n=97)$.}
\end{figure}
@@ -300,15 +301,15 @@
\item FCI gain versus gain in prominence of solution-oriented and physics-related postings
\end{itemize}
-As it turns out, the first correlations are significant, with $R=-0.44 [-0.65 \to -0.18] (n=47)$ for FCI gain versus solution-oriented discussions, and $R=0.4 [0.13 \to 0.62] (n=47)$ for FCI gain versus physics-related discussions. Such significant correlations do not occur for FCI gain versus any of the MPEX cluster scores.
+As it turns out, the first correlations are significant, with $R=-0.44\ [-0.65\to-0.18] (n=47)$ for FCI gain versus solution-oriented discussions, and $R=0.4\ [0.13\to0.62] (n=47)$ for FCI gain versus physics-related discussions. Such significant correlations do not occur for FCI gain versus any of the MPEX cluster scores.
-On the other hand, the correlations with discussion-gain are not significant: $0.24 [-0.05 \to 0.49] (n=47)$ for FCI gain versus gain in solution-oriented discussions, and $-0.12 [-0.39 \to 0.17] (n=47)$ for FCI gain versus gain in physics-related discussions. Note that these correlations have the opposite sign than expected, however, the confidence intervals include zero in both cases. When looking at the absolute values, the average gain in solution-oriented discussions between the two halves of the semester is $2.4\%$, and the gain in physics-oriented discussions $-0.3\%$ --- in other words, the students did not really change their discussion behavior over the course of the semester, and their discussion behavior does not improve co-measured with their increasing understanding of physics.
+On the other hand, the correlations with discussion-gain are not significant: $0.24\ [-0.05\to0.49] (n=47)$ for FCI gain versus gain in solution-oriented discussions, and $-0.12\ [-0.39\to0.17] (n=47)$ for FCI gain versus gain in physics-related discussions. Note that these correlations have the opposite sign than expected, however, the confidence intervals include zero in both cases. When looking at the absolute values, the average gain in solution-oriented discussions between the two halves of the semester is $2.4\%$, and the gain in physics-oriented discussions $-0.3\%$ --- in other words, the students did not really change their discussion behavior over the course of the semester, and their discussion behavior does not improve co-measured with their increasing understanding of physics.
Thus, the discussion behavior appears to be a property of the students that is almost constant over the course of the semester, just like Hammer~\cite{hammer94} already pointed out that it is unlikely that epistemological beliefs are changed implicitly by physics instruction.
-We also ran a linear regression analysis of the FCI scores versus discussion behavior. In the equations below, ``PostFCI'' is the predicted post (final) FCI score, ``PreFCI'' is the score on the pre FCI, and ``Solution'' and ``Physics'' are the percentage solution- and physics-oriented discussion over the course of the semester. For the physics-oriented discussion, we found
+We also ran a linear regression analysis of the FCI scores versus discussion behavior. In the equations below, ``PostFCI'' is the predicted post (final) FCI score in points, ``PreFCI'' is the score on the pre-test FCI in points, and ``Solution'' and ``Physics'' are the percentage solution-oriented and physics-related discussion over the course of the semester. For the physics-oriented discussion, we found
\begin{equation*}
-\mbox{Post FCI}=5.486+0.922\cdot\mbox{PreFCI}+0.24\cdot\mbox{Physics}
+\mbox{PostFCI}=5.486+0.922\cdot\mbox{PreFCI}+0.24\cdot\mbox{Physics}
\end{equation*}
with an explained variance of 45.6\% of the Post FCI score. The effect of the pre-test FCI is significant ($p<0.001$), the effect of the physics discussion is not ($p=0.195$).
@@ -316,7 +317,7 @@
\begin{equation*}
\mbox{PostFCI}=7.606+0.857\cdot\mbox{PreFCI}+(-0.042)\cdot\mbox{Solution}
\end{equation*}
-with an explained variance of 47.9\% of the Post FCI score. Both coefficients are significant, the solution-oriented discussion has $p=0.019$. Thus, controlling for Pre FCI score, for each 10 percent increase in solution-oriented discussion, the predicted Post FCI score goes down by 0.42 points.
+with an explained variance of 47.9\% of the Post FCI score. Both coefficients are significant, the solution-oriented discussion has $p=0.019$. Thus, controlling for pre-test FCI score, for each 10 percent increase in solution-oriented discussion, the predicted post-test FCI score goes down by 0.42 points. Students who do not make any solution-oriented contributions would on the average gain 7.6 points on the 30 item FCI due to instruction, while at the other extreme, students who only make solution-oriented discussions would on the average only gain 3.4 points -- less than half.
\section{Conclusions}
Online student discussions have very little correlation with MPEX outcomes, but appear to be a good reflection of students' individual beliefs regarding the nature of problem solving in physics. Students who exhibit more expert-like views and strategies have higher learning success, even when controlling for prior physics knowledge.
\begin{acknowledgments}
--www1159659948--