[LON-CAPA-cvs] cvs: modules /gerd/concept description.tex
www
lon-capa-cvs@mail.lon-capa.org
Wed, 14 Jul 2004 21:16:30 -0000
This is a MIME encoded message
--www1089839790
Content-Type: text/plain
www Wed Jul 14 17:16:30 2004 EDT
Modified files:
/modules/gerd/concept description.tex
Log:
Great progress.
--www1089839790
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20040714171630.txt"
Index: modules/gerd/concept/description.tex
diff -u modules/gerd/concept/description.tex:1.12 modules/gerd/concept/description.tex:1.13
--- modules/gerd/concept/description.tex:1.12 Wed Jul 14 15:59:11 2004
+++ modules/gerd/concept/description.tex Wed Jul 14 17:16:30 2004
@@ -55,17 +55,17 @@
\subsection{Overview}
This five-year project focusses on online formative assessment in introductory physics education, and how formative assessment can be used to help learners re-evaluate their epistemologies, develop expertlike problem solving skills, and gain a conceptual understanding of physics. It will compare the impact of online problems which are categorized across 21 types previously identified in literature.
-The study will be carried out in on-campus courses with regular classroom times, which are enhanced by online components. The NSF-supported Learning{\it Online} Network with Computer-Assisted Personalized Approach (LON-CAPA) will be used as the model system.
+The study will be carried out in on-campus courses with regular classroom times, which are enhanced by online components. The NSF-supported Learning{\it Online} Network with Computer-Assisted Personalized Approach (LON-CAPA; Section~\ref{loncapa}) will be used as the model system.
The project has three components:
\begin{enumerate}
-\item new development, as well as adaptation and implementation of research-based problems inside of LON-CAPA, where the currently existing library of problems does not have a sufficient number of representatives of this type
+\item new development, as well as adaptation and implementation of research-based problems inside of LON-CAPA, where the currently existing library of problems does not have a sufficient number of representatives of this type (Section~\ref{matdev})
\item additional tool development inside of LON-CAPA to provide
\begin{itemize}
-\item higher scalability for deploying problem types that cannot be completely evaluated by the computer
-\item better analysis tools for the purpose of this study
+\item higher scalability for deploying problem types that cannot be completely evaluated by the computer (Section~\ref{platform})
+\item better analysis tools for the purpose of this study (Section~\ref{analysisnew})
\end{itemize}
-\item hypotheses testing for each problem type regarding their educational impact
+\item hypotheses testing for each problem type regarding their educational impact (Section~\ref{analysis})
\end{enumerate}
\subsection{"Thinking like a Physicist"}
@@ -200,9 +200,11 @@
\end{figure}
\subsection{Courses}
-The project will be carried out in the two-semester LBS course sequence LBS 271/272,"Calculus-Based Introductory Physics I/II. These second-year three-credit courses have a Calculus pre-requisite, and traditionally an enrollment of over 200 students. Two separate, but associated one-credit laboratory courses (LBS 271L/272L) are required, which most but not all students choose to take simultaneously.
+The project will be carried out in the two-semester LBS course sequence LBS 271/272,"Calculus-Based Introductory Physics I/II. These second-year three-credit courses have a Calculus pre-requisite, and traditionally an enrollment of over 200 students.
-Faculty and teaching assistants are frequently assuming shared responsibilities between the lecture and laboratory courses, with a combined staff of two faculty members and six undergraduate student assistants. The latter are responsible for particular recitation and laboratory sections, and will be involved in this research project (see Section~\ref{undergrad}). Within the duration of this project, the lecture and laboratory courses might be combined to provide greater coherence between these two venues.
+Starting Fall 2004, the course will be taught with less total lecturing time, where the third classroom hour will be used for peer-teaching~\cite{mazur} and more frequent summative assessment in place of the midterm exams.
+
+Two separate, but associated one-credit laboratory courses (LBS 271L/272L) are required, which most but not all students choose to take simultaneously. Faculty and teaching assistants are frequently assuming shared responsibilities between the lecture and laboratory courses, with a combined staff of two faculty members and six undergraduate student assistants. The latter are responsible for particular recitation and laboratory sections, and will be involved in this research project (see Section~\ref{undergrad}). Within the duration of this project, the lecture and laboratory courses might be combined to provide greater coherence between these two venues.
Students in these courses are currently solving approximately 250 online homework problems each semester, most of which currently are of the conventional type.
\subsection{Synergy between Project and Institutional Goals}
@@ -269,28 +271,33 @@
\section{Preliminary Project Components}
Several aspects of the research project can be started in the first year, while others will require additional functionality in the LON-CAPA platform, and the preparation of additional homework problems of certain types.
-In Table~\ref{classification}, problem types which are marked "S" with no additional symbols have both sufficient functionality support in the LON-CAPA system, and a sufficient library of problems of this type to conduct the study. If a problem type is marked "M," the library of problems of this type is still small, and additional problems need to be developed. If a problem type is marked "A," it measn that the study would profit from the development of additional analysis tools, if it is marked "G," it means that while the platform in its current version can support it, the solution does not scale well with the number of students due to the effort required for manual grading.
+In Table~\ref{classification}, problem types which are marked "S" with no additional symbols have both sufficient functionality support in the LON-CAPA system, and a sufficient library of problems of this type to conduct the study. If a problem type is marked "M," the library of problems of this type is still small, and additional problems need to be developed. If a problem type is marked "A," it means that the study would profit from the development of additional analysis tools, if it is marked "G," it means that while the platform in its current version can support it, the solution does not scale well with the number of students due to the effort required for manual grading.
+
+\subsection{Additional Materials Development}\label{matdev}
+For the question types marked "M" in Table~\ref{classification}, the currently existing library of LON-CAPA problems is not provide enough samples to carry out the study. Within this project, new homework problems of this type will be developed. Since development of completely new problems would constitute a project by itself, this component of the current project will heavily draw on existing problem collections, i.e., Redish (\cite{redish}, resource CD), McDermott~\cite{mcdermottprob}, Mazur~\cite{mazur}, and Project Galileo~\cite{galileo}. These research-based problems will be adapted and implemented in the the LON-CAPA system.
+
\subsection{Additional Platform Development}
+
+\subsubsection{Scalable Functionality for Manual Grading of Free-Form Answers}\label{platform}
+LON-CAPA already offers grading support for free-form student submission, such as keyword highlighting and plagiarism-checks. Additional tools will be developed for the grading of the problem types marked "G" in Table~\ref{classification}: for questions that require student submissions of the type "Explain your reasoning," better coupling between the computer- and manually-evaluated sections will be provided, for the free-form "essay" submissions better tools to compare student submissions with each other and with examplary essays.
+
\subsubsection{Additional Analysis Tools}\label{analysisnew}
While the premise of this project is that feedback on formative assessment is crucial for the learner, it is almost equally important to the instructor~\cite{pellegrino}, with technology as enabler~\cite{novak,feedback}. Particularly in the context of a research project on formative assessement, timely and comprehensive feedback on student performance --- including new material (section~\ref{matdev}) --- is essential. The LON-CAPA system already has sophisticated analysis tools (see section~\ref{analysis}), but these do not support all questions types in Table~\ref{classification} equally well, and the project includes a tools development component to further enhance these mechanisms for the problem types marked "A."
Data collection on a particular problem type can proceed independently from the existence of the respective analysis tools, since LON-CAPA permanently stores all data.
-\subsubsection{Scalable Functionality for Manual Grading of Free-Form Answers}\label{platform}
-LON-CAPA already offers grading support for free-form student submission, such as keyword highlighting and plagiarism-checks. Additional tools will be developed for the grading of the problem types marked "G" in Table~\ref{classification}: for questions that require student submissions of the type "Explain your reasoning," better coupling between the computer- and manually-evaluated sections will be provided, for the free-form "essay" submissions better tools to compare student submissions with each other and with examplary essays.
-\subsection{Additional Materials Development}\label{matdev}
-For the question types marked "M" in Table~\ref{classification}, the currently existing library of LON-CAPA problems is not provide enough samples to carry out the study. Within this project, new homework problems of this type will be developed. Since development of completely new problems would constitute a project by itself, this component of the current project will heavily draw on existing problem collections, i.e., Redish (\cite{redish}, resource CD), McDermott~\cite{mcdermottprob}, Mazur~\cite{mazur}, and Project Galileo~\cite{galileo}. These research-based problems will be adapted and implemented in the the LON-CAPA system.
+
\section{Research Methodology}\label{analysis}
\subsection{Establishment of Initial Conditions}The validity of the hypotheses may depend on both learner and assessment characteristics.\subsubsection{Learner Attitudes, Beliefs, and Expectations}Instruments have been developed to assess epistemological beliefs, for example the Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument~\cite{EBAPS}. Related to epistemological beliefs are learnerÕs expectations and attitudes, and of particular interest is the Maryland Physics Expectations (MPEX) survey~\cite{MPEX}.\subsubsection{Learner Knowledge about the Topic}\label{prepost}We will use existing concept inventory surveys as both pre- and post-tests.The qualitative Force Concept Inventory~\cite{fci} and the quantitative companion Mechanical Baseline Test~\cite{hestenesmech} have been used in a large number of studies connected to the teaching of introductory mechanics. The Foundation Coalition has been developing a number of relevant concept inventories~\cite{foundation}, namely the Thermodynamics Concept Inventory, the Dynamics Concept Inventory, and the Electromagnetics Concept Inventory (with two subcomponents, namely Waves and Fields). Since these were designed from an engineering point of view, some adjustment might be necessary. In addition, the Conceptual Survey of Electricity and Magnetism (CSEM)~\cite{maloney} is available for the second semester course.
\subsubsection{Problem Difficulty and Baseline Statistical Data}LON-CAPA automatically keeps tracks of the average number of attempts until a problem is solved, as well as the degree of difficulty and the degree of discrimination. This data is cumulative across semesters, and already exists for all assessment problems from their deployment in previous semesters.
\subsection{Observables}\subsubsection{Effectiveness}Effectiveness will be measured both in terms of performance on summative assessments (quizzes and exams) and on pre-/post-test concept inventory surveys (Section~\ref{prepost}). Each item on these instruments will be associated with topically corresponding formative online exercises to determine correlations and differential gain between the feedback types used with the respective online problems. A second posttest, correlated with first semester problems, will be administered at the end of the second semester to determine long-term effects.\subsubsection{Problem Solving Technique}We intend to focus on a subset of students in the LBS Collaborative Learning Laboratory, and observe them while solving problems. Schoenfeld~\cite{schoenfeld} and Foster~\cite{foster} developed instruments to categorize and document the stages and expertlike
-characteristics~\cite{chi} of observed problem-solving activity by learners, as well as application of metacognitive skills.In addition, we will interview a group of students from all courses regarding their problem-solving strategies. Pascarella~\cite{pascarella02} developed some frameworks for these interviews, which can be built upon.Finally, for all students in all courses, LON-CAPA log data will be analyzed. Kotas~\cite{kotas} and Minaei~\cite{minaei} developed a mechanism for this log data analysis, which include submission times between attempts, and quality of the entered input. \subsubsection{Help-Seeking Behavior}It is impossible to observe all on-demand help seeking, but interactions in several settings can be analyzed:Online discussions and email communication are preserved within LON-CAPA and can be analyzed even in retrospect for past semesters with respect to relevant behavioral patterns. Table~\ref{discussion} shows excerpts of discussions around the two problems in Fig.~\cite{trajectories}.
+characteristics~\cite{chi} of observed problem-solving activity by learners, as well as application of metacognitive skills.In addition, we will interview a group of students from all courses regarding their problem-solving strategies. Pascarella~\cite{pascarella02} developed some frameworks for these interviews, which can be built upon.Finally, for all students in all courses, LON-CAPA log data will be analyzed. Kotas~\cite{kotas} and Minaei~\cite{minaei} developed a mechanism for this log data analysis, which include submission times between attempts, and quality of the entered input. \subsubsection{Help-Seeking Behavior}\label{discussion}It is impossible to observe all on-demand help seeking, but interactions in several settings can be analyzed:Online discussions and email communication are preserved within LON-CAPA and can be analyzed even in retrospect for past semesters with respect to relevant behavioral patterns. Table~\ref{discussionex} shows excerpts of discussions around the two problems in Fig.~\cite{trajectory}.
\begin{table}
\tiny
-\begin{tabular}{p{3.2in}|p{3.2in}}
+\begin{tabular}{p{3.1in}|p{3.1in}}
{\bf Student A:}
since your not given the initial velocity or the angle,
but you know the distance covered, couldnt the angle be
@@ -439,11 +446,11 @@
degrees I got it wrong. Be VERY careful when drawing the
tangent.
\end{tabular}
-\caption{Excepts from online discussion around the two problems Fig.~\ref{trajectories}}
+\caption{Excepts from online discussion around the two problems Fig.~\ref{trajectory}\label{discussionex}}
\end{table}
-Discussion contributions and states can be linked to online transactions, such as submission of homework attempts, browsing of content material, and hint usage. Wallace~\cite{wallace} reviewed existing research on such online interactions, however, some adaptation of several of the existing conceptualizations will be necessary to account for the nature of physics courses.For the subset of LBS students who come to the LBS Collaborative Learning Laboratory, group discussions can to be documented, and linked to online behavior as absolute timing and learner identify are preserved.Minaei~\cite{minaei} developed data mining strategies to categorize learner behavior, including navigational patterns between assessment and content material, the use of feedback, and communication functions.Self-reporting can be used for several other help-seeking mechanisms, such as textbook use and peer-interaction~\cite{riffell1,riffell2}.\subsubsection{Cross-Cutting Open-Ended Documentation of Learner Perceptions}We will interview a focus groups of students regarding their experiences and perceived relative helpfulness of the different problem types, and ask them to also reflect on how they perceived these question types were influencing their problem-solving strategies.
-\subsection{Data Collection and Comparison Groups}For the data collection, results would be most meaningful, if we divided the course into groups that experience the same assignment with different feedback mechanisms; indeed, this is what we propose to do in later phases of the study regarding the variation of solely the feedback character. However, when varying the number of available tries and the immediacy of the feedback, we will do so concurrently for all students across different problems Š otherwise, it would be inevitable that students will complain about unfairness compared to the students with more attempts or immediate feedback. Also students with delayed feedback would most certainly confer with students in the other group.
+Discussion contributions and states can be linked to online transactions, such as submission of homework attempts, browsing of content material, and hint usage. Wallace~\cite{wallace} reviewed existing research on such online interactions, however, some adaptation of several of the existing conceptualizations will be necessary to account for the nature of physics courses.For the subset of LBS students who come to the LBS Collaborative Learning Laboratory, group discussions can to be documented, and linked to online behavior as absolute timing and learner identify are preserved.Minaei~\cite{minaei} developed data mining strategies to categorize learner behavior, including navigational patterns between assessment and content material, the use of feedback, and communication functions.Self-reporting can be used for several other help-seeking mechanisms, such as textbook use and peer-interaction~\cite{riffell1,riffell2}.\subsubsection{Cross-Cutting Open-Ended Documentation of Learner Perceptions}We will interview focus groups of students regarding their experiences and perceived relative helpfulness of the different problem types, and ask them to also reflect on how they perceived these question types were influencing their problem-solving strategies.
+\subsection{Data Collection and Comparison Groups}
\section{Involvement of Undergraduate Students in Research}\label{undergrad}
@@ -452,7 +459,27 @@
\section{Dissemination}
We will present papers at conferences such as the LON-CAPA User Conference, Frontiers in Education, and the American Association of Physics Teachers Annual Meeting. We will submit papers to journals such as The Physics Teacher, the American Journal of Physics, Computers and Education, and the Journal of Asynchronous Learning Networks. Finally, any content material adapted and implemented in this project will be immediately available to all participant LON-CAPA institutions, and via the LON-CAPA gateway to the NSF-funded National Science Digital Library.
\section{Project Timeline}
+The timeline for the project is outlined in Table~\ref{timeline}.
+The first year will be used as control group using the existing 250 problems/semester, while both tool and materials development are ongoing.
+\begin{table}
+\begin{tabular}{|l|p{3.2cm}|p{3.2cm}|p{3.2cm}|p{3.2cm}|}
+\hline
+&Tool Development&Materials Development&Study&Other\\
+\hline
+Year 1&\\
+\hline
+Year 2&\\
+\hline
+Year 3&\\
+\hline
+Year 4&\\
+\hline
+Year 5&\\
+\hline
+\end{tabular}
+\caption{Project timeline\label{timeline}}
+\end{table}
\section{Results from Prior NSF Support}\label{results}
@@ -479,7 +506,7 @@
\bibitem{steinberg} Richard Steinberg, {\it Performance on Multiple-Choice Diagnostics and Complementary Exam Problems}, Phys. Teach {\bf 35}(3), 150-155.
\bibitem{mazur} Eric Mazur, {\it Peer Instruction}, Prentice Hall, ISBN 0-13-565441-b (1997)
\bibitem{chi} Michelene T. H. Chi, Paul J. Feltovich, Robert Glaser, {\it Categorization and Representation of Physics Problems by Experts and Novices}, Cognitive Science, Vol 5, p121-152 (1981)
-\bibitem{larkin} J. Larkin, J. McDermott, D. P. Simon, and H. A. Simon, {\it Expert and novice performance in solving physics problems}, Science {\bf 208} 1335-1342 (1980)
+\bibitem{larkin} J. Larkin, J. McDermott, D. P. Simon, and H. A. Simon, {\it Expert and novice performance in solving physics problems}, Science {\bf 208}, 1335-1342 (1980)
\bibitem{reif} Frederick Reif, {\it Teaching problem solving --- A scientific approach}, Phys. Teach. {\bf 19}, 310-316 (1981)
\bibitem{mazur96} Eric Mazur, {\it The Problem with Problems}, Optics and Photonics News {\bf 6}, 59-60 (1996)
\bibitem{hammer} David Hammer, {\it More than misconceptions: Multiple perspectives on student knowledge and reasoning, and an appropriate role for education research}, Am. J. Phys. {\bf 64}, 1316-1325 (1996)
@@ -487,7 +514,7 @@
\bibitem{kashy00} E. Kashy, M. Thoennessen Y. Tsai, N. E. Davis, and G. Albertelli II, {\it Melding Network Technology with Traditional Teaching: Enhanced Achievement in a 500-Student Course}, Chapter in {\it Interactive Learning: Vignettes from America's Most Wired Campuses},
David G. Brown (editor), Anker Publishing Company, Boston, 51, ISBN 1-882982-29-0 (2000)
\bibitem{hewitt} Paul G. Hewitt, {\it Conceptual Physics}, Little, Brown, ISBN 0-673-39541-3
-\bibitem{tobias} Sheila Tobias, {\it Overcoming Math Anxiety}, W. W. Norton\& Company; Revised edition, ISBN 0-393-31307-7 (1995)
+\bibitem{tobias} Sheila Tobias, {\it Overcoming Math Anxiety}, W. W. Norton \& Company; Revised edition, ISBN 0-393-31307-7 (1995)
\bibitem{tobiasST} Sheila Tobias, {\it They're not dumb, they're different. Stalking the Second Tier}, Research Corporation (1990)
\bibitem{torigoe} Eugene Torigoe, {\it Student Difficulties with Equations in Physics}, ISAAPT Spring Meeting, Urbana, IL, (April 2004)
\bibitem{breitenberger} Ernst Breitenberger, {\it The mathematical knowledge of physics graduates: Primary data and conclusions}, Am. J. Phys. {\bf 60}(4), 318-323 (1992)
@@ -523,14 +550,14 @@
\bibitem{EBAPS} A. Elby, J. Fredriksen, C. Schwarz, and B. White, {\it Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument}, {\tt http://www2.physics.umd.edu/$\sim$elby/EBAPS/EBAPS\_items.htm} (2001)
\bibitem{MPEX} E. F. Redish, J. M. Saul, and R. N. Steinberg, {\it Student expectations in introductory physics}, Am. J. Physics {\bf 66}, 212-224 (1998), survey available on The Physics Suite CD in Teaching Physics, ISBN 0-471-39378-9 (2003)
\bibitem{hestenesmech} D. Hestenes and M.Wells, {\it A Mechanics Baseline Test}, Phys. Teach {\bf 30}(3), 159-166 (1992)
-\bibitem{foundation} Foundation Coalition, {\it Key Components: Concept Inventories}, {\tt http://www.foundationcoalition.org/home/keycomponents/concept/index.html} (2003)
+\bibitem{foundation} Foundation Coalition, {\it Key Components: Concept Inventories},\newline {\tt http://www.foundationcoalition.org/home/keycomponents/concept/index.html} (2003)
\bibitem{maloney} D. P. Maloney, T. L. O$'$Kuma, C. J. Hieggelke, A. van Heuvelen, {\it Surveying studentsÕ conceptual knowledge of electricity and magnetism}, Am. J. Phys. Suppl. {\bf 69}, S12 (2001)
\bibitem{schoenfeld} A. H. Schoenfeld, {\it Mathematical Problem Solving}, Academic Press (1985)
\bibitem{foster} T. M. Foster, {\it The Development of StudentÕs Problem-Solving Skill from Instruction Emphasizing Qualitative Problem-Solving}, dissertation, University of Minnesota (2000)
\bibitem{minaei} Behrouz Minaei-Bidgoli, William. F. Punch, {\it Using Genetic Algorithms for Data Mining Optimization in an Educational Web-based System}, Proceedings, Genetic and Evolutionary Computation Conference (2003)
\bibitem{kotas} P. Kotas, Homework Behavior in an Introductory Physics Course, Masters Thesis (Physics), Central Michigan University (2000)
\bibitem{wallace} Raven M. Wallace, {\it Online Learning in Higher Education: a review of research on interactions among teachers and students}, Education, Communication and Information,
-{\bf 3}(2), 241 (2003)\bibitem{riffell1} Samuel K. Riffell and Duncan F. Sibley, {\it Can hybrid course formats increase attendance in undergraduate environmental science courses?}, Journal of Natural Resources and Life Sciences Education, in press (2003) \bibitem{riffell2} Samuel K. Riffell and Duncan F. Sibley, D.F., {\it Student perceptions of a hybrid learning format: can online exercises replace traditional lectures?}, Journal of College Science Teaching, {\bf 32}(6), 394-399 (2003)
+{\bf 3}(2), 241 (2003)\bibitem{riffell1} Samuel K. Riffell and Duncan F. Sibley, {\it Can hybrid course formats increase attendance in undergraduate environmental science courses?}, Journal of Natural Resources and Life Sciences Education, in press (2003) \bibitem{riffell2} Samuel K. Riffell and Duncan F. Sibley, {\it Student perceptions of a hybrid learning format: can online exercises replace traditional lectures?}, Journal of College Science Teaching, {\bf 32}(6), 394-399 (2003)
\end{thebibliography}
\end{document}
\end
\ No newline at end of file
--www1089839790--