# [LON-CAPA-cvs] cvs: modules /gerd/correlpaper correlations.tex

www lon-capa-cvs@mail.lon-capa.org
Wed, 12 Jul 2006 02:10:09 -0000

This is a MIME encoded message

--www1152670209
Content-Type: text/plain

www		Tue Jul 11 22:10:09 2006 EDT

Modified files:
/modules/gerd/correlpaper	correlations.tex
Log:
First round of corrections

--www1152670209
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20060711221009.txt"

Index: modules/gerd/correlpaper/correlations.tex
diff -u modules/gerd/correlpaper/correlations.tex:1.4 modules/gerd/correlpaper/correlations.tex:1.5
--- modules/gerd/correlpaper/correlations.tex:1.4	Tue Jul 11 14:18:24 2006
+++ modules/gerd/correlpaper/correlations.tex	Tue Jul 11 22:10:06 2006
@@ -51,7 +51,7 @@
%  but any date may be explicitly specified

\begin{abstract}
-An important result of Physics Education Research is that students' learning and success in a course is correlated with their beliefs, attitudes, and expectations. In this study, we investigate the possibility of using the evaluation of online student discussion behavior as an indicator of a student's approach to physics. We investigate correlations with a traditional instrument, namely the Maryland Physics Expectations Survey (MPEX), as well as correlations with the Force Concept Inventory (FCI), the final exam grade, and the overall course performance as a measure of the student's learning. To gauge the outcomes, we also investigate correlations between these measures.\end{abstract}
+An important result of Physics Education Research is that students' learning and success in a course is correlated with their beliefs, attitudes, and expectations regarding physics. In this study, we investigate the possibility of using the analysis of online student discussion behavior as an indicator of a student's approach to physics. We investigate correlations with a traditional instrument, namely the Maryland Physics Expectations Survey (MPEX), as well as correlations with the Force Concept Inventory (FCI), the final exam grade, and the overall course performance as a measure of the student's learning. To gauge the outcomes, we also investigate correlations between these measures.\end{abstract}

\pacs{01.40.Fk}% PACS, the Physics and Astronomy
% Classification Scheme.
@@ -63,16 +63,16 @@

Online student discussions associated with online physics problems are different in that they are generated within the real context of the course, and students have a vested interest in making these discussions as productive as possible, given their understanding of how physics is done and their approach to it. They could thus be a reality check'' of students' beliefs, attitudes, and expectations.

-Online discussions are a rich source of feedback to the instructor~\cite{kortemeyer05feedback}, and their quality and character was found to be correlated with the type and difficulty of the associated problems~\cite{kortemeyer05ana}, i.e., data exists regarding the influence of {\it problem} characteristics on associated discussions. Unfortunately, less data exists on the correlation between {\it student} characteristics and discussion behavior, because only very few student characteristics were known, such as the students' overall performance in the course. One of the findings was the fact that certain discussion behavior, most prominently exhibited on non-sanctioned'' discussion sites external to the course, is negatively correlated with performance in the course~\cite{kashy03,kortemeyer05ana}.
+Online discussions are a rich source of feedback to the instructor~\cite{kortemeyer05feedback}, and their quality and character was found to be correlated with the type and difficulty of the associated problems~\cite{kortemeyer05ana}, i.e., data exists regarding the influence of {\it problem} characteristics on associated discussions. Unfortunately, less data exists on the correlation between {\it student} characteristics and discussion behavior, because usually only very few student characteristics are known, with the exception of the students' overall performance in the course. Thus, one of the few findings was the fact that certain discussion behavior, most prominently exhibited on non-sanctioned'' discussion sites external to the course, is negatively correlated with performance in the course~\cite{kashy03,kortemeyer05ana}.

-In this study, we aim to answer the question if and how student discussion characteristics are related to their beliefs and attitudes as measured by the MPEX. We investigate correlations with the MPEX, and compare correlations with measures of student learning.
+In this study, we aim to answer the question if and how student discussion characteristics are related to their beliefs, attitudes, and expectations, as measured by the MPEX. We investigate correlations with the MPEX, and compare correlations with measures of student learning.

\section{\label{setting}Setting}
-The project was carried out in an introductory calculus-based physics course with initially 214 students. Most of the students in this course plan on pursuing a career in a medical field. The course had three traditional lectures per week. It did not use a textbook, instead, all course materials were available online. Topics were introductory mechanics, as well as sound and thermodynamics. There was twice-weekly online homework: one small set as reading problems due before the topic was dealt with in class (implementing JiTT~\cite{jitt}), and a larger set of traditional end-of-the-chapter style homework at the end of each topic. The problems in the course were randomized using the LON-CAPA system, i.e., different students would receive different versions of the same problem (different graphs, numbers, images, options, formulas, etc)~\cite{loncapa,kashyd01}. The students had weekly recitation sessions, and a traditional lab was offered in parallel. The course grade was determined from the students' performance on the biweekly quizzes, the final exam, the recitation grades, and the homework performance.
+The project was carried out in an introductory calculus-based physics course with initially 214 students. Most of the students in this course plan on pursuing a career in a medical field. The course had three traditional lectures per week. It did not use a textbook, instead, all course materials were available online. Topics were introductory mechanics, as well as sound and thermodynamics. There was twice-weekly online homework: one small set as reading problems due before the topic was dealt with in class (implementing JiTT~\cite{jitt}), and a larger set of traditional end-of-the-chapter style homework at the end of each topic. The problems in the course were randomized using the LON-CAPA system, i.e., different students would receive different versions of the same problem (different graphs, numbers, images, options, formulas, etc)~\cite{loncapa,kashyd01}. The students had weekly recitation sessions, and a traditional lab was offered in parallel. The course grade was determined from the students' performance on biweekly quizzes, the final exam, the recitation grades, and the homework performance.

\section{\label{measures}Measures and Instruments}
\subsection{\label{discussion}Discussion Analysis}
-We also analyzed the online student discussions that were associated with the online homework given in the course, using the scheme first suggested in Ref.~\cite{kortemeyer05ana}. There were a total of 2405 such online discussion contributions over the course of the semester.
+We analyzed the online student discussions that were associated with the online homework given in the course, using the scheme first suggested in Ref.~\cite{kortemeyer05ana}. There were a total of 2405 such online discussion contributions over the course of the semester.

Each contribution was classified according to the classification scheme of Ref.~\cite{kortemeyer05ana}, however, with the additional refinement that each contribution could be member of more than one class, and that the contributions were weighted by their length. For example, a certain contribution might include both a procedural solution-oriented question and a surface-level mathematical answer, and would thus receive 50\% membership in both classes, weighted by its total length.

@@ -173,28 +173,28 @@
\end{itemize}
Note that correctness of the contribution was not considered. For example, in the last physics-related example, the fact that the student confused the gravitational constant and the gravitational acceleration was not taken into account.

-As already found in Ref.~\cite{kortemeyer05ana}, most students are quite prolific in their online discussions, but a few students only made a small number of contributions, leading to small statistics on their actual discussion behavior. For each of the discussion correlations, we thus carried out a second calculation limited to students who contributed at least five entries over the course of the semester.
+As already found in Ref.~\cite{kortemeyer05ana}, most students are quite prolific in their online discussions, but a few students only made a small number of contributions, leading to small statistics on their actual discussion behavior. For each of the discussion correlations, we thus also carried out a second calculation limited to students who contributed at least five entries over the course of the semester.

\subsection{\label{mpex}The MPEX}
-We deployed the Maryland Physics Expectations Survey (MPEX)\cite{mpex} both at the beginning and the end of the mechanics semester. Participation was voluntary. We calculated the score in comparison to the favorable" expert responses given in Ref.~\cite{mpex} on the final deployment, as well as, for students who participated both times, the gain. The same analysis was done for the clusters of the MPEX (example statements are given, including the expert answer):
+We deployed the Maryland Physics Expectations Survey (MPEX)\cite{mpex} both at the beginning and the end of the mechanics semester. Participation was voluntary. We calculated the score in comparison to the favorable" expert responses given in Ref.~\cite{mpex} on the final (post) deployment, as well as, for students who participated both times, the gain. The same analysis was done for each cluster of the MPEX (example statements are given, including the expert answer):
\begin{itemize}
-\item{\it Independence:} student takes responsibility for constructing their own understanding rather than takes what is given by authorities (teacher, materials) without evaluation
+\item{\it Independence:} student takes responsibility for constructing their own understanding, rather than takes what is given by authorities (teacher, materials) without evaluation
\begin{quote}
Unfavorable: In this course, I do not expect to understand equations in an intuitive sense; they must just be taken as givens.
\end{quote}
-\item{\it Coherence:} student believes that physics needs to be considered as a connected, consistent framework rather than a set of unrelated facts or pieces''
+\item{\it Coherence:} student believes that physics needs to be considered as a connected, consistent framework, rather than a set of unrelated facts or pieces''
\begin{quote}
Unfavorable: Knowledge in physics consists of many pieces of information each of which applies primarily to a specific situation.
\end{quote}
-\item{\it Concepts:} student stresses the understanding of the underlying ideas and concepts rather than the memorization and usage of formulas
+\item{\it Concepts:} student stresses the understanding of the underlying ideas and concepts, rather than the memorization and usage of formulas
\begin{quote}
Favorable: When I solve most exam or homework problems, I explicitly think about the concepts that underlie the problem.
\end{quote}
-\item{\it Reality Link:} student believes that the ideas learned in physics are relevant and useful in a wide variety of real contexts rather than having little to do with experiences outside the classroom
+\item{\it Reality Link:} student believes that the ideas learned in physics are relevant and useful in a wide variety of real contexts, rather than having little to do with experiences outside the classroom
\begin{quote}
Unfavorable: Physical laws have little relation to what I experience in the real world.
\end{quote}
-\item{\it Math Link:} student considers mathematics as a convenient way of representing physical phenomena rather than viewing physics and mathematics as having little or no relationship
+\item{\it Math Link:} student considers mathematics as a convenient way of representing physical phenomena, rather than viewing physics and mathematics as having little or no relationship
\begin{quote}
Unfavorable: All I learn from a derivation or proof of a formula is that the formula obtained is valid and that it is OK to use it in problems.
\end{quote}
@@ -259,13 +259,13 @@
MPEX &    & &&     &       &     &           & & & & & \\
Gain   &   0.27 &0.26 &   ---       & ---         &          &         &---               (---)&---      (---)&---         (0.18)&0.14     (---)&--0.14   (---)&---       (---)\\
Indepen- &    & &&     &       &     &           & & & & & \\
-dence&0.25 &0.25&0.23      &0.1       &          &         &---               (---)&---   (0.13)&0.12         (---)&---        (---)&---       (---)&---    (0.15)\\
+dence&0.25 &0.25&0.23      &0.1       &          &         &---               (---)&---   (0.13)&---         (---)&---        (---)&---       (---)&---    (0.15)\\
Cohe- &    & &&     &       &     &           & & & & & \\
-rence   &0.36 &0.31&0.2       &0.23      &          &         &---               (---)&---   (0.14)&0.18      (0.15)&---        (---)&---    (0.12)&0.1     (---)\\
+rence   &0.36 &0.31&0.2       &0.23      &          &         &---               (---)&---   (0.11)&0.18      (0.18)&---        (--0.18)&---    (0.24)&0.1     (0.17)\\
Con-&    & &&     &       &     &           & & & & & \\
-cepts    &0.25 &0.24 &0.21      &0.13      &          &         &---               (---)&---      (---)&0.11      (0.14)&---        (---)&---       (---)&0.14 (0.18)\\
+cepts    &0.25 &0.24 &0.21      &0.13      &          &         &---               (---)&---      (---)&0.11      (0.14)&---        (---)&---       (---)&0.14 (0.15)\\
Reality &    & &&     &       &     &           & & & & & \\
-Link&---    &0.1&0.15      &---         &          &         &--0.12           (---)&---      (---)&---           (---)&--0.2 (--0.14)&0.11 (0.15)&---       (---)\\
+Link&---    &0.1&0.15      &---         &          &         &--0.12           (---)&---      (--0.16)&---           (---)&--0.2 (---)&0.11 (---)&---       (---)\\
Math &    & &&     &       &     &           & & & & & \\
Link   &0.15 &---&0.13      &---         &          &         &---               (---)&---  (--0.14)&--- (---)               &0.1      (---)&---       (---)&---       (---)\\
Effort      &0.15&0.15  &---         &0.11      &          &         &--0.14       (--0.19)&---      (---)&0.19       (0.2)&--0.14 (--0.12)&---       (---)&0.22 (0.15)
@@ -274,7 +274,7 @@
\end{ruledtabular}
\end{table*}

-Of particular interest is the lower right corner of Table~\ref{fullresults}, as it lists the correlations between student attitudes and expectations (as measured by the MPEX clusters) with the prominence of discussion behavior classes. One would have expected strong correlations between for example the score on the Concepts Cluster and the prominence of conceptual discussion contributions ($R=0.14 (0.18)$), or the comfort level with the usage of mathematics as a language and the corresponding lack of purely mathematical contributions (non significant, and $R=-0.14$ when including only students with more than five contributions overall).
+Of particular interest is the lower right corner of Table~\ref{fullresults}, as it lists the correlations between student attitudes and expectations (as measured by the MPEX clusters) with the prominence of discussion behavior classes. One would have expected strong correlations between for example the score on the Concepts Cluster and the prominence of conceptual discussion contributions ($R=0.14 (0.15); n=84 (51)$), or the comfort level with the usage of mathematics as a language and the corresponding lack of purely mathematical contributions (non significant, and $R=-0.14$ ($n=51$) when including only students with more than five contributions overall). The Coherence and Effort Clusters are most strongly correlated with discussions, the Math Link Cluster -- surprisingly -- the least.

The upper right and the lower left corner list the correlations of student discussion behavior and the MPEX, respectively, with measures of student learning. Correlations are again low, but of comparable magnitude, where the MPEX appears to be slightly more correlated with grade and final exam performance, while the discussion is more correlated with the FCI. In fact, some of the strongest correlations in the study occur between the prominence of solution-oriented and physics-related discussions and the FCI. We will analyze correlations with grades in more detail in subsection~\ref{gradecorrel}, and with the FCI in subsection~\ref{fcicorrel}.

@@ -286,15 +286,15 @@

-Figure~\ref{fcimpexgrade} shows the correlation between the final FCI and MPEX scores with the final course grade percentage. With an $r$ of 0.56 and 0.30, respectively, these -- particularly for the MPEX -- turned out lower than expected. As pointed out in section~\ref{setting}, however, the course grade is based on a number of factors, some of which are simply a matter of diligence or effort.
+Figure~\ref{fcimpexgrade} shows the correlation between the final FCI and MPEX scores with the final course grade percentage. With an $R$ of 0.56 ($n=110$) and 0.30 ($n=97$), respectively, these -- particularly for the MPEX -- turned out lower than expected. As pointed out in section~\ref{setting}, however, the course grade is based on a number of factors, some of which are simply a matter of diligence or effort.

\begin{figure*}
\caption{\label{fcimpexgrade}Correlation between the FCI score (left; $R=0.56$; $n=110$) and the MPEX score (right; $R=0.3$; $n=97$) with the course grade percentage. 58\% was the minimum percentage to pass the course. More students participated in the FCI than in the MPEX.}
\end{figure*}

-Correlations with the MPEX were generally weak, with $R=0.36$ between the score on the Coherence cluster and the course grade percentage being the highest value. Dancy~\cite{dancy02} found similarly low correlations with the performance on homework, tests, and final exams: direct comparison with the performance on the final exams found $R=0.37$ for the correlation with the total MPEX score ($R=0.27$ here), $R=0.39$ with the Independence Cluster ($R=0.25$ here), $R=0.24$ with the Coherence Cluster ($R=0.36$ here), $R=0.29$ with the Concept Cluster ($R=0.25$ here), $R=-.02$ with the Reality Link cluster ($R=0.1$ here), $R=0.3$ with the Math Link cluster (no significant correlation found here), and no significant correlation with the Effort Cluster ($R=0.1$ here).
+Correlations with the MPEX were generally weak, with $R=0.36$ ($n=97$) between the score on the Coherence cluster and the course grade percentage being the highest value. Dancy~\cite{dancy02} found similarly low correlations with the performance on homework, tests, and final exams: direct comparison with the performance on the final exams found $R=0.37$ for the correlation with the total MPEX score ($R=0.27$ here), $R=0.39$ with the Independence Cluster ($R=0.25$ here), $R=0.24$ with the Coherence Cluster ($R=0.36$ here), $R=0.29$ with the Concept Cluster ($R=0.25$ here), $R=-.02$ with the Reality Link cluster ($R=0.1$ here), $R=0.3$ with the Math Link cluster (no significant correlation found here), and no significant correlation with the Effort Cluster ($R=0.1$ here).

Figure~\ref{physicsgrade} shows the correlation between the prominence of physics-related discussions and the course grade percentage (for better statistics, only students who contributed at least five discussion entries over the course of the semester were considered). The correlation is stronger than with the MPEX Score, yet smaller than with the FCI.
\begin{figure}
@@ -305,7 +305,7 @@
\subsection{\label{fcicorrel}Correlations with the FCI}
Figure~\ref{mpexfci} shows how the final FCI and MPEX scores correlated with each other, i.e, $R=0.24$ ($n=97$).
Coletta and Philips~\cite{coletta05} found a strong correlation between the FCI Gain and the MPEX Score ($R=0.52; n=37$), while the same correlation turned out much lower in this study ($R=0.17; n=84$ here). The correlations reported here are in the same range that
-Perkins et al.~\cite{perkins04} found when investigating the influence of beliefs on conceptual learning, using the CLASS~\cite{adams04} and the Force and Motion Conceptual Evaluation (FMCE) instruments~\cite{thornton98}.
+Perkins et al.~\cite{perkins04} found when investigating the influence of beliefs on conceptual learning, using the CLASS~\cite{adams04} and the Force and Motion Conceptual Evaluation (FMCE)~\cite{thornton98} instruments.

Correlations with discussion characteristics turned out somewhat stronger. Figure~\ref{fciphysics} shows how the percentage of a particular student's discussion contribution that was classified as "physics-related" correlates with their final FCI score  ($R=0.34$; $n= 95$). As already in Fig.~\ref{physicsgrade}, an additional analysis was carried out that was limited to students for which better statistics were available, which let to a stronger correlation ($R=0.51$; $n=57$).

@@ -318,7 +318,7 @@
\includegraphics[width=9cm]{fcipostphysics}\includegraphics[width=9cm]{fcipostphysicsT}
\caption{\label{fciphysics}Correlation between the FCI score and the percentage of that student's discussion that was classified as "physics" ($R=0.34$; $n= 95$). The figure on the right only includes students who contributed more than five discussion entries over the course of the semester ($R=0.51$; $n=57$).}
\end{figure*}
-While physics-related discussions positively correlate with FCI scores and grades (Fig.~\ref{physicsgrade}), solution-oriented discussions negatively correlated (Fig.~\ref{solutionfci}; $R=-0.58$; $n=57$).
+While physics-related discussions positively correlate with FCI scores and grades (Fig.~\ref{physicsgrade}), solution-oriented discussions negatively correlate (Fig.~\ref{solutionfci}; $R=-0.58$; $n=57$).
\begin{figure}
\includegraphics[width=9cm]{fcipostsolutionT}
\caption{\label{solutionfci}Correlation of percentage solution-oriented discussions with final FCI score ($R=-0.58$; $n=57$).}
@@ -328,7 +328,7 @@

An unexpected result were the low correlations between the MPEX cluster scores and the student discussion behavior. We can thus not conclude that student discussion behavior is strongly correlated with student attitudes and expectations as measured by the MPEX. Student discussions and the MPEX also differently correlate to measures of learning, i.e., student discussion more strongly correlates to the FCI, and MPEX more strongly to course grades and the final exam.

-The relative weakness of many of the expected correlations with the MPEX might indicate that maybe -- in spite of the efforts of the author -- the students did not take the MPEX very seriously or did not carefully read the statements. An argument for this possible explanation is that the overall scores of the students on the MPEX were low (Independence 42\%; Coherence 46\%; Concepts 48\%; Reality Link 55\%; Math Link 40\%; Effort 47\%). Also, students relatively frequently chose the answer "3" ("Neutral") on the MPEX Likert scale, which is by definition never correct --- answering that way could indicate true indifference, confusion regarding the statement, or simply "don't care." By the same token, students appear to be taking the FCI more seriously, probably because it more closely matches the other (grade-relevant) assessments they encounter in the course, and students tend to based their relative value system regarding a subject area on the assessments used~\cite{lin}. The FCI seems to be fairly robust in ungraded settings, see for example Henderson~\cite{henderson}, who found only 0.5 points difference between graded and ungraded administration of the FCI --- the MPEX, which is never graded, may in fact be far less robust to perception of  not counting."
+The relative weakness of many of the expected correlations with the MPEX might indicate that maybe -- in spite of the efforts of the author -- the students did not take the MPEX very seriously or did not carefully read the statements. An argument for this possible explanation is that the overall scores of the students on the MPEX were low (Independence 42\%; Coherence 46\%; Concepts 48\%; Reality Link 55\%; Math Link 40\%; Effort 47\%). Also, students relatively frequently chose the answer "3" ("Neutral") on the MPEX Likert scale, which is by definition never correct --- answering that way could indicate true indifference, or confusion regarding the statement, or simply "don't care." By the same token, students appear to be taking the FCI more seriously, probably because it more closely matches the other (grade-relevant) assessments they encounter in the course, and students tend to based their relative value system regarding a subject area on the assessments used~\cite{lin}. The FCI seems to be fairly robust in ungraded settings, see for example Henderson~\cite{henderson}, who found only 0.5 points difference between graded and ungraded administration of the FCI --- the MPEX, which is never graded, may in fact be far less robust to the perception of  not counting."

\section{Conclusions}
In this introductory calculus-based course, correlations between different performance and attitude indicators were found to be lower than expected. Student discussion behavior generally correlates more strongly with student performance (FCI, final exam, grade) than MPEX results. Particularly the prominence of solution-oriented and physics-related discussions correlate relatively strongly with the FCI.

--www1152670209--