[LON-CAPA-cvs] cvs: modules /gerd/concept description.tex

[www]

This is a MIME encoded message

--www1089925992
Content-Type: text/plain

www		Thu Jul 15 17:13:12 2004 EDT

  Modified files:              
    /modules/gerd/concept	description.tex 
  Log:
  Wonderful.
  
  
--www1089925992
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20040715171312.txt"

Index: modules/gerd/concept/description.tex
diff -u modules/gerd/concept/description.tex:1.15 modules/gerd/concept/description.tex:1.16
--- modules/gerd/concept/description.tex:1.15	Thu Jul 15 11:05:20 2004
+++ modules/gerd/concept/description.tex	Thu Jul 15 17:13:11 2004
@@ -52,7 +52,7 @@
 \end{quote}
 
 \subsection{Overview}
-This five-year project focusses on online formative assessment in introductory physics education, and how it can be used to help learners re-evaluate their epistemologies, develop expertlike problem solving skills, and gain a conceptual understanding of physics. It will compare the impact of online problems which are categorized across 21 types previously identified in literature. 
+This five-year project focuses on online formative assessment in introductory physics education, and how it can be used to help learners re-evaluate their epistemologies, develop expertlike problem solving skills, and gain a conceptual understanding of physics. It will compare the impact of online problems which are categorized across 21 types previously identified in literature. 
 
 The study will be carried out in on-campus courses with regular classroom times, which are enhanced by online components. The NSF-supported Learning{\it Online} Network with Computer-Assisted Personalized Approach (LON-CAPA; Sect.~\ref{loncapa}) will be used as the model system.
 
@@ -64,7 +64,7 @@
 \item higher scalability for deploying problem types that cannot be completely evaluated by the computer (Sect.~\ref{platform})
 \item better analysis tools for the purpose of this study (Sect.~\ref{analysisnew})
 \end{itemize}
-\item hypotheses-testing for each problem type regarding their educational impact (Sect.~\ref{analysis})
+\item hypotheses-testing for each problem type regarding its educational impact (Sect.~\ref{analysis})
 \end{enumerate}
 
 \subsection{"Thinking like a Physicist"}
@@ -120,7 +120,7 @@
 \begin{quote}
 {\bf Hypothesis 1b:} There are types of online formative assessment computer-evaluated problems which make learners confront their non-expertlike epistemologies and encourage expertlike problem-solving strategies\end{quote}
 
-These problem types might involve both computer- and human-evaluated components, where an emphasis has to be put on keeping the human-evaluated part managable and scalable.
+These problem types might involve both computer- and human-evaluated components, where an emphasis has to be put on keeping the human-evaluated part manageable and scalable.
 
 An additional problem with conventional problems may be their mathematical nature. Hewitt in the preface to his textbook "Conceptual Physics"~\cite{hewitt} argues that the mathematical language of physics often deters the average non-science students, a notion which concurs with Tobias' concept of "math anxiety"\cite{tobias}, which is a particular issue for students in the "second tier"\cite{tobiasST} of science courses.  For them, the use of mathematics in physics courses can present a hurdle, and a lack of skills or confidence to perform basic algebraic manipulations ("$V=RI\ \Rightarrow\ R=V/I"$), or even problems operating their pocket calculators, can hinder students' learning progress in physics at a very basic level.
 
@@ -140,15 +140,15 @@
 \begin{quote}
 {\bf Hypothesis 3:} Learners with an average or above average level of mathematical skills or confidence will more likely develop a conceptual understanding of physics as a result of non-calculation-oriented online formative assessment, by discouraging non-expertlike problem-solving strategies.
 \end{quote}
-As evidenced in the above definition, it should be emphasized that the project does by no means attempt to establish or promote a dichotomomy between "conceptual understanding" and "basic skills/factual knowledge." A physicist needs basic skills and factual knowledge, and the learning of these must not be underemphasized in formative assessment. However, how to best develop these through formative assessment would constitute another valid research project.
+As evidenced in the above definition, it should be emphasized that the project does by no means attempt to establish or promote a dichotomy between "conceptual understanding" and "basic skills/factual knowledge." A physicist needs basic skills and factual knowledge, and the learning of these must not be underemphasized in formative assessment. However, how to best develop these through formative assessment would constitute another valid research project.
 \subsection{Intellectual Merit}
 Online homework is becoming increasingly prominent in physics education, yet research into its effect has been contradictory, sparse, and, in some cases, not very systematic~\cite{pasc04}. Differences have been reported positive~\cite{kashyda}, negative~\cite{pasc04}, and non-significant~\cite{bonham}. Pascarella~\cite{pascarella02} was one of the very few studies investigating problem-solving strategies, but both Pascarella~\cite{pascarella02} and Bonham~\cite{bonham} only considered the online versions of conventional textbook-like problems. Kashy~\cite{kashyd01b} found that what the authors call "interactive problems," namely those where learners need to read relevant values from graphs or observe simulations, are better predictors of overall success in the course than other problem types, but did not investigate cause and effect relationships, or study problem-solving behavior. 
 
-This study aims to provide a systematic research base regarding the effectiveness of different types of online formative assessment, especially those which do take better advantage of the medium, and inform both curriculum development efforts and practicioners.
+This study aims to provide a systematic research base regarding the effectiveness of different types of online formative assessment, especially those which do take better advantage of the medium, and inform both curriculum development efforts and practitioners.
 \subsection{Broader Impact/Diversity}
 Currently, every semester approximately 350,000 students are taking introductory undergraduate physics courses similar to the ones under investigation in this project~\cite{aapt}. For many of these students, it is both their first and their last formal exposure to physics. Students will go into a large spectrum of careers, with or without an understanding of the basic concepts of the physical world.
 
-This project has the potential of broader impact, since like many of the other efforts in Physics Education, it is closely connected to the trenches of physics teaching. Results from this study will be applicable in courses across the nation, especially large-enrollment courses. Both the tool (LON-CAPA, Sect.~\ref{loncapa}) and any developed, implemented, and adapted materials (Sect.~\ref{matdev}) will be readily available to physics faculty. Faculty members at the over thirty currenly participating LON-CAPA institutions will be able to profit from this project already during its progress.
+This project has the potential of broader impact, since like many of the other efforts in Physics Education, it is closely connected to the trenches of physics teaching. Results from this study will be applicable in courses across the nation, especially large-enrollment courses. Both the tool (LON-CAPA, Sect.~\ref{loncapa}) and any developed, implemented, and adapted materials (Sect.~\ref{matdev}) will be readily available to physics faculty. Faculty members at the over thirty currently participating LON-CAPA institutions will be able to profit from this project already during its progress.
 
 \section{Background and Environment}
 \subsection{PI Education and Appointments}
@@ -282,21 +282,21 @@
 
 \subsubsection{Scalable Functionality for Manual Grading of Free-Form Answers}\label{platform}
 LON-CAPA already offers grading support for free-form student submission, such as 
-keyword-highlighting and plagiarism-checks. Additional tools will be developed for the grading of the problem types marked "G" in Table~\ref{classification}: for questions that require student submissions of the type "Explain your reasoning," better coupling between the computer- and manually-evaluated sections will be provided, for the free-form "essay" submissions, better tools to compare student submissions with each other and with examplary essays.
+keyword-highlighting and plagiarism-checks. Additional tools will be developed for the grading of the problem types marked "G" in Table~\ref{classification}: for questions that require student submissions of the type "Explain your reasoning," better coupling between the computer- and manually-evaluated sections will be provided, for the free-form "essay" submissions, better tools to compare student submissions with each other and with exemplary essays.
 
 \subsubsection{Additional Analysis Tools}\label{analysisnew}
-While the premise of this project is that feedback on formative assessment is crucial for the learner, it is almost equally important to the instructor~\cite{pellegrino}, with technology as enabler~\cite{novak,feedback}. Particularly in the context of a research project on formative assessement, timely and comprehensive feedback on student performance --- including new material (Sect.~\ref{matdev}) --- is essential. The LON-CAPA system already has sophisticated analysis tools (see Sect.~\ref{anatool}), but these do not support all questions types in Table~\ref{classification} equally well, and the project includes a tools development component to further enhance these mechanisms for the problem types marked "A."
+While the premise of this project is that feedback on formative assessment is crucial for the learner, it is almost equally important to the instructor~\cite{pellegrino}, with technology as enabler~\cite{novak,feedback}. Particularly in the context of a research project on formative assessment, timely and comprehensive feedback on student performance --- including new material (Sect.~\ref{matdev}) --- is essential. The LON-CAPA system already has sophisticated analysis tools (see Sect.~\ref{anatool}), but these do not support all questions types in Table~\ref{classification} equally well, and the project includes a tools development component to further enhance these mechanisms for the problem types marked "A."
 
 Data collection on a particular problem type can proceed independently  from the existence of the respective analysis tools, since LON-CAPA permanently stores all data.
 
 
 
 \section{Research Methodology}\label{analysis}
-\subsection{Establishment of Initial Conditions}The validity of the hypotheses may depend on both learner and assessment characteristics.\subsubsection{Learner Attitudes, Beliefs, and Expectations}Instruments have been developed to assess epistemological beliefs, for example the Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument~\cite{EBAPS}. Related to epistemological beliefs are learnerÕs expectations and attitudes, and of particular interest is the Maryland Physics Expectations (MPEX) survey~\cite{MPEX}.\subsubsection{Learner Knowledge about the Topic}\label{prepost}We will use existing concept inventory surveys as both pre- and post-tests.The qualitative Force Concept Inventory~\cite{fci} and the quantitative companion Mechanical Baseline Test~\cite{hestenesmech} have been used in a large number of studies connected to the teaching of introductory mechanics. The Foundation Coalition has been developing a number of relevant concept inventories~\cite{foundation}, namely the Thermodynamics Concept Inventory, the Dynamics Concept Inventory, and the Electromagnetics Concept Inventory (with two subcomponents, namely Waves and Fields).  Since these were designed from an engineering point of view, some adjustment might be necessary. In addition, the Conceptual Survey of Electricity and Magnetism (CSEM)~\cite{maloney} is available for the second semester course.
+\subsection{Establishment of Initial Conditions}The validity of the hypotheses may depend on both learner and assessment characteristics other than their type.\subsubsection{Learner Attitudes, Beliefs, and Expectations}Instruments have been developed to assess epistemological beliefs, for example the Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument~\cite{EBAPS}. Related to epistemological beliefs are learnerÕs expectations and attitudes, and of particular interest is the Maryland Physics Expectations (MPEX) survey~\cite{MPEX}.\subsubsection{Learner Knowledge about the Topic}\label{prepost}We will use existing concept inventory surveys as both pre- and post-tests.The qualitative Force Concept Inventory~\cite{fci} and the quantitative companion Mechanical Baseline Test~\cite{hestenesmech} have been used in a large number of studies connected to the teaching of introductory mechanics. The Foundation Coalition has been developing a number of relevant concept inventories~\cite{foundation}, namely the Thermodynamics Concept Inventory, the Dynamics Concept Inventory, and the Electromagnetics Concept Inventory (with two subcomponents, namely Waves and Fields).  Since these were designed from an engineering point of view, some adjustment might be necessary. In addition, the Conceptual Survey of Electricity and Magnetism (CSEM)~\cite{maloney} is available for the second semester course.
 
 \subsubsection{Problem Difficulty and Baseline Statistical Data}LON-CAPA automatically keeps tracks of the average number of attempts until a problem is solved, as well as the degree of difficulty and the degree of discrimination. This data is cumulative across semesters, and already exists for all assessment problems from their deployment in previous semesters.
 \subsection{Observables}\subsubsection{Effectiveness}Effectiveness will be measured both in terms of performance on summative assessments (quizzes and exams) and on pre-/post-test concept inventory surveys (Sect.~\ref{prepost}).  Each item on these instruments will be associated with topically corresponding formative online exercises to determine correlations and differential gain between the feedback types used with the respective online problems. A second posttest, correlated with first semester problems, will be administered at the end of the second semester to determine long-term effects.\subsubsection{Problem Solving Technique}We intend to focus on a subset of students in the LBS Collaborative Learning Laboratory, and observe them while solving problems. Schoenfeld~\cite{schoenfeld} and Foster~\cite{foster} developed instruments to categorize and document the stages and expertlike 
-characteristics~\cite{chi} of observed problem-solving activity by learners, as well as application of metacognitive skills.In addition, we will interview a group of students from all courses regarding their problem-solving strategies. Pascarella~\cite{pascarella02} developed some frameworks for these interviews, which can be built upon.Finally, for all students in all courses, LON-CAPA log data will be analyzed. Kotas~\cite{kotas} and Minaei~\cite{minaei} developed a mechanism for this log data analysis, which include submission times between attempts, and quality of the entered input. \subsubsection{Help-Seeking Behavior}\label{discussion}It is impossible to observe all on-demand help seeking, but interactions in several settings can be analyzed:Online discussions and email communication are preserved within LON-CAPA and can be analyzed even in retrospect for past semesters with respect to relevant behavioral patterns.  Table~\ref{discussionex} shows excerpts of discussions around the two problems in Fig.~\cite{trajectory}.
+characteristics~\cite{chi} of observed problem-solving activity by learners, as well as application of metacognitive skills.In addition, we will interview a group of students from all courses regarding their problem-solving strategies. Pascarella~\cite{pascarella02} developed some frameworks for these interviews, which can be built upon.Finally, for all students in all courses, LON-CAPA log data will be analyzed. Kotas~\cite{kotas} and Minaei~\cite{minaei} developed a mechanism for this log data analysis, which include submission times between attempts, and quality of the entered input. \subsubsection{Help-Seeking Behavior and Discussions}\label{discussion}It is impossible to observe all on-demand help seeking, but interactions in several settings can be analyzed:Online discussions and email communication are preserved within LON-CAPA and can be analyzed even in retrospect for past semesters with respect to relevant behavioral patterns.  Table~\ref{discussionex} shows excerpts of discussions around the two problems in Fig.~\ref{trajectory}.
 
 \begin{table}
 \tiny
@@ -449,13 +449,17 @@
 degrees I got it wrong. Be VERY careful when drawing the 
 tangent.
 \end{tabular}
-\caption{Excepts from online discussion around the two problems Fig.~\ref{trajectory}\label{discussionex}}
+\caption{Excerpts from online discussion around the two problems Fig.~\ref{trajectory}\label{discussionex}}
 \end{table}
 
 Discussion contributions and states can be linked to online transactions, such as submission of homework attempts, browsing of content material, and hint usage. Wallace~\cite{wallace} reviewed existing research on such online interactions, however, some adaptation of several of the existing conceptualizations will be necessary to account for the nature of physics courses.For the subset of LBS students who come to the LBS Collaborative Learning Laboratory, group discussions can to be documented, and linked to online behavior as absolute timing and learner identify are preserved.Minaei~\cite{minaei} developed data mining strategies to categorize learner behavior, including navigational patterns between assessment and content material, the use of feedback, and communication functions.Self-reporting can be used for several other help-seeking mechanisms, such as textbook use and peer-interaction~\cite{riffell1,riffell2}.\subsubsection{Cross-Cutting Open-Ended Documentation of Learner Perceptions}We will interview focus groups of students regarding their experiences and perceived relative helpfulness of the different problem types, and ask them to also reflect on how they perceived these question types were influencing their problem-solving strategies. 
-\subsection{Data Collection and Comparison Groups}
+
+\section{Involvement of Students in Research}
+\subsection{Undergraduate}\label{undergrad}
+Students in the courses will be informed about the goals and methods of this project beyond the requirements of the consent procedures. Groups of students will be given the opportunity to participate in focus groups, for which stipends are provided in the budget. The undergraduate teaching assistants in this course will be given the option to participate in the data collection and analysis efforts of the project. Research findings will be made available to the students.
+\subsection{Graduate}
+The proposal budget includes funds for a graduate student assistantship to to assist in the materials development effort, as well as to collect and analyze data, and work on publication and dissemination. It is expected that the student will base his or her doctoral work on this project. The idea of offering a doctoral degree in Science Education has been considered at MSU, yet at this point in time, the degree would be conferred by the Department of Physics and Astronomy. The doctoral committee for the student will be comprised of members of the Department of Physics and Astronomy and the Division of Science and Mathematics Education.
 
-\section{Involvement of Undergraduate Students in Research}\label{undergrad}
 
 \section{Evaluation}
 The LON-CAPA Faculty Advisory Board was formed as part of our NSF ITR grant project. It consists of eight actively teaching faculty and administrators from a number of colleges on campus of MSU, and meets once every month to both evaluate and advise projects connected to LON-CAPA. We propose to continue using this existing structure to evaluate this projectÕs progress and findings. In addition, Dr.~Kortemeyer's Mentoring Committee, which consists of senior faculty members from both LBS and DSME will guide and advise the progress of this project.
@@ -463,22 +467,26 @@
 We will present papers at conferences such as the LON-CAPA User Conference, Frontiers in Education, and the American Association of Physics Teachers Annual Meeting.  We will submit papers to journals such as The Physics Teacher, the American Journal of Physics, Computers and Education, and the Journal of Asynchronous Learning Networks.  Finally, any content material adapted and implemented in this project will be immediately available to all participant LON-CAPA institutions, and via the LON-CAPA gateway to the NSF-funded National Science Digital Library.
 \section{Project Timeline}
 The timeline for the project is outlined in Table~\ref{timeline}.
-
-The first year will be used as control group using the existing 250 problems/semester, while both tool and materials development are ongoing.
+.
 \begin{table}
 \begin{tabular}{|l|p{3.2cm}|p{3.2cm}|p{3.2cm}|p{3.2cm}|}
 \hline
 &Tool Development&Materials Development&Study&Other\\
 \hline
-Year 1&\\
-\hline
-Year 2&\\
+Year 1&Grading Tools&Categorization of the approx. 400 applicable problems per semester according to question type\newline
+Adaptation and implementation of new problems
+&Pre- and post-tests\newline
+Review and refinement of research methodology&Recruitment of programmer and graduate student\newline
+Visits to other Physics Education Research groups
+\\
 \hline
-Year 3&\\
+Year 2&Grading Tools&Adaptation, implementation, and development of new problems
+\newline
+First deployment of new problems&First deployment of research methods and iterative refinement&Visits to other Physics Education Research groups\\
 \hline
-Year 4&\\
+Years 3 and 4&Grading and Analysis Tools&Refinement of new problems&Data collection and analysis&Dissemination\\
 \hline
-Year 5&\\
+Year 5&&&Final data analysis&Dissemination\\
 \hline
 \end{tabular}
 \caption{Project timeline\label{timeline}}

--www1089925992--