[LON-CAPA-cvs] cvs: modules /gerd/concept description.tex

[www]

This is a MIME encoded message

--www1089300964
Content-Type: text/plain

www		Thu Jul  8 11:36:04 2004 EDT

  Modified files:              
    /modules/gerd/concept	description.tex 
  Log:
  Work on dissemination, eval, and methodology
  
  
--www1089300964
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20040708113604.txt"

Index: modules/gerd/concept/description.tex
diff -u modules/gerd/concept/description.tex:1.5 modules/gerd/concept/description.tex:1.6
--- modules/gerd/concept/description.tex:1.5	Wed Jul  7 21:25:33 2004
+++ modules/gerd/concept/description.tex	Thu Jul  8 11:36:03 2004
@@ -21,8 +21,7 @@
 \topmargin = 0.0 in
 \headheight = 0.0 in
 \headsep = 0.3 in
-\parskip = 0.1in
-\parindent = 0.0in
+
 \pagestyle{headings}
 
 \begin{document}
@@ -31,9 +30,9 @@
 \else
 	\DeclareGraphicsExtensions{.eps, .jpg}
 \fi
-\begin{title}
-\title{\LARGE\sc Physics education:\\ Does "conceptual" online formative assessment lead to conceptual understanding?}
-\end{title}
+\begin{center}
+\LARGE\sc Physics education:\\ Does "conceptual" online formative assessment lead to conceptual understanding?
+\end{center}
 
 \section{Introduction}
 \begin{quote}
@@ -53,19 +52,19 @@
 \end{quote}
 
 \subsection{"Thinking like a Physicist"}
-Most physicists would agree that part of "being a physicist" is to "think like a physicist" \cite{heuvelen}, which manifests itself most prominently in the cognitive and metacognitive skills involved in problem solving~\cite{redish} --- part of the self-image of many physicists is to be expert problem solvers, even outside their own discipline \cite{fuller}:
+Most physicists would agree that part of "being a physicist" is to "think like a physicist"~\cite{heuvelen}, which manifests itself most prominently in the cognitive and metacognitive skills involved in problem solving~\cite{redish} --- part of the self-image of many physicists is to be expert problem solvers, even outside their own discipline~\cite{fuller}:
 \[\mbox{Think like a Physicist}\ \Rightarrow\ \mbox{Solve Problems}\]
 Yet, while basic logic tells us that the reverse statement, i.e., 
 \[\mbox{Think like a Physicist}\ \Leftarrow\ \mbox{Solve Problems}\]
 does not necessarily have to be true,
-an unfortunately only too common approach to teaching physics is to have students solve a lot of problems. 
+an unfortunately only too common approach to teaching physics is to simply have students solve a lot of standard problems. 
 
-It is broadly accepted that frequent formative assessment and feedback are a key component of the learning process \cite{bransford}, but {\it what} the students are learning is not necessarily what educators are expecting them to learn \cite{pellegrino}. This disconnect frequently goes undetected if the summative assessment tools are similar to the formative assessment tools \cite{lin}. In most physics courses, these are standard problems along the lines of "A ball starts with an initial velocity of \ldots. \ldots What is the \ldots?". Deploying alternative conceptual assessment tools such as the Force Concept Inventory \cite{fci} can reveal large discrepancies between different summative assessment types (e.g,~\cite{steinberg,mazur}).
+It is broadly accepted that frequent formative assessment and feedback are a key component of the learning process~\cite{bransford}, but {\it what} the students are learning is not necessarily what educators are expecting them to learn~\cite{pellegrino}. This disconnect frequently goes undetected if the summative assessment tools are similar to the formative assessment tools~\cite{lin}. In most physics courses, these are standard problems along the lines of "A ball starts with an initial velocity of \ldots. \ldots What is the \ldots?". Deploying alternative conceptual assessment tools such as the Force Concept Inventory~\cite{fci} can reveal large discrepancies between different summative assessment types (e.g,~\cite{steinberg,mazur}).
 
 Expert and novice approaches to problem solving in physics have been studied extensively (e.g.~\cite{chi,larkin}). Two of the most apparent differences are that
 \begin{enumerate}
 \item experts are initially characterizing problems according to deep structure and physical concepts (e.g., "energy conservation"-problem), while novices tend to characterize them according to surface features (e.g., "sliding-block-on-incline"-problem) or applicable formulas (e.g., "$E=\frac12mv^2+mgh$"-problem")
-\item novices then continue to employ a formula-centered problem solving method \cite{heuvelen}, frequently refered to as "plug-and-chug."
+\item novices then continue to employ a formula-centered problem solving method~\cite{heuvelen}, frequently refered to as "plug-and-chug."
 \end{enumerate}
 Redish~\cite{redish} somewhat bleakly  describes a novice approach to learning physics as follows:
 \begin{itemize}
@@ -76,23 +75,25 @@
 \item Erase all information from your brain after the exam to make room for the next set of materials.
 \end{itemize}
 
-One cannot really blame learners for shortcircuiting physics "learning" this way, since the cognitive and metacognitive skills, which physicists value so much higher than factual knowledge or formulas, are hardly ever made explicit, neither in instruction, nor in formative or summative assessment \cite{lin,reif,mazur96}; in fact, they are mostly altogether "hidden" \cite{redish} from all aspects of a course, and students are affirmed in their novice expectations \cite{hammer} of what it is to "do physics."
+One cannot really blame learners for shortcircuiting physics "learning" this way, since the cognitive and metacognitive skills, which physicists value so much higher than factual knowledge or formulas, are hardly ever made explicit, neither in instruction, nor in formative or summative assessment~\cite{lin,reif,mazur96}; in fact, they are mostly altogether "hidden"~\cite{redish} from all aspects of a course, and students are affirmed in their novice expectations~\cite{hammer} of what it is to "do physics."
 \subsection{The Problem with Problems}
-{\bf This project focusses on formative assessment in introductory physics education, and how formative assessment can be used to help learners re-evaluate their epistemologies and develop expert-like problem solving skills.} The challenge is to move students away from treating physics as a set of unrelated factoids and formulas, and away from focussing on memorizing and using formulas without interpretation or sense-making \cite{hammer}.
+{\bf This project focusses on formative assessment in introductory physics education, and how formative assessment can be used to help learners re-evaluate their epistemologies and develop expert-like problem solving skills.} The challenge is to move students away from treating physics as a set of unrelated factoids and formulas, and away from focussing on memorizing and using formulas without interpretation or sense-making~\cite{hammer}.
 
-Alternative formative assessment as a classroom tool, where students are forced to verbally  express their views and teach each other, rather than calculate answers \cite{mazur}, is starting to be accepted as an effective teaching practice in more and more courses. Formative assessment outside the classroom, on the other hand, is frequently limited by logistics, particularly in large enrollment courses, where timely feedback is often impossible without the use of computerized systems (e.g.~\cite{thoennessen,kashy00}). {\bf This project will focus on online homework as formative assessment tool.}
+Alternative formative assessment as a classroom tool, where students are forced to verbally  express their views and teach each other, rather than calculate answers~\cite{mazur}, is starting to be accepted as an effective teaching practice in more and more courses. Formative assessment outside the classroom, on the other hand, is frequently limited by logistics, particularly in large enrollment courses, where timely feedback is often impossible without the use of computerized systems (e.g.~\cite{thoennessen,kashy00}). {\bf This project will focus on online homework as formative assessment tool.} As a model system, the project will use the Learning{\it Online} Network with CAPA (LON-CAPA), described in section~\ref{loncapa}.
 
-The assumption of this project is that "the problem with problems" (a phrase borrowed from \cite{mazur96}) is that
+The assumption of this project is that "the problem with problems" (a phrase borrowed from~\cite{mazur96}) is that
 \begin{quote}
-Hypothesis 1a: Standard textbook problems affirm non-expertlike epistemologies and encourage non-expertlike problem-solving strategies
+Hypothesis 1a: Standard calculation-oriented textbook problems affirm non-expertlike epistemologies and encourage non-expertlike problem-solving strategies
 \end{quote}
 and that by the reverse token --- within the limitations of online computer-evaluated problems ---
 \begin{quote}
-Hypothesis 1b: There are classes of online formative assessment computer-evaluated problems which make learners confront their non-expertlike epistemologies and encourage expertlike problem-solving strategies\end{quote}
+Hypothesis 1b: There are classes of online formative assessment computer-evaluated problems which make learners confront their non-expertlike epistemologies and encourage expert-like problem-solving strategies\end{quote}
 
 Online homework systems by the very nature of computers lend themselves to standard calculation-oriented textbook problems, and are extensively used in this way. Yet, the "plug-and-chug" approach is the most prominent symptom of novice-like problem-solving strategy, and calculation-oriented problems may encourage just that. As a result, there is a frequent call for "conceptual" online problems, where both instructors and students seem to define "conceptual" simply by the absence of numbers and formulas. But does depriving students of numbers and formulas indeed make them work on a conceptual level?
 
-The project will study the use and abuse of mathematics in formative assessment.
+Hewitt in the preface to his textbook "Conceptual Physics"~\cite{hewitt} argues that the mathematical language of physics often deters the average nonscience students, a notion which concurs with Tobias' concept of "math phobia"[CITE]. For many students, though, the opposite appears to be true, their "concept phobia" appears to be more prominent than any "math phobia."
+
+
 
 
 
@@ -101,38 +102,49 @@
 
 Moving beyond these initial barriers, students are generally able to correctly substitute variables and execute calculations, but see formulas in a purely operational sense~\cite{torigoe}, while lacking the abilitity to translate between the formulas and the situations~\cite{clement}. 
 
-\subsection{Formulas versus "Conceptual"}
-Most frequently, the conflict can be boiled down to the use or abuse of formulas: A trait of expert physicists is that  also for them a lot of formulas might have been "erased from the brain," but can be reconstructed from the knowledge of underlying concepts, as well as the cognitive and metacognitive processes needed to make the connections. While experts see mathematics as a means to express complex concepts, novices tend to see them as a mechanism to "plug-and-chug." 
+\subsection{Intellectual Merit}Much effort and many resources have been invested into developing effective curricular material and assessment, especially in the interactive or online realm, yet very little research has been done on the impact of different representations and question types on student conceptual understanding.The outcomes of this study will provide a broader research base for STEM curriculum development efforts regarding the most effective use of learner feedback. The outcomes will also inform development efforts for online course and learning content management systems, as well as provide input for educational metadata, content exchange, and interoperability standard efforts.\subsection{Broader Impact/Diversity}Attrition in STEM courses is a national problem and may be particularly acute in large, introductory courses that are the staples of many undergraduate science programs, with 40 to 60\% of students leaving STEM disciplines [Committee99, Seymour97]. Much of this attrition has been attributed to inadequate curriculum design, pedagogy and assessment.It is broadly accepted that frequent formative assessment and feedback are a key component of the learning process~\cite{bransford}. Shifting the focus from summative to formative assessment with feedback can move student motivation from an extrinsic reward to the intrinsic reward of developing understanding of the materials~\cite{stipek}. Intrinsic motivation and positive feedback promote the feelings of competence, confidence~\cite{clark,pascarella02}, and engagement that are crucial to retaining students in introductory STEM courses.  While improving student self-efficacy should have positive impacts on all student retention, Seymour and Hewitt [Seymour97] suggest that such changes should have a particularly strong impact on women and underrepresented groups who may feel that science excludes them.
+
+
+
+They report that even women with good academic records nonetheless lost confidence in judging their academic performances as "good enough" if they did not receive feedback through personal teacher-student relationships Ð these relationships were reported to be rare, possibly due to the disproportionate number of male teaching staff in STEM courses. Lack of confidence, paired with a perceived gender or race bias, can negatively influence performance [Steele97].A gender-differential benefit of using online formative assessment systems (particularly LON-CAPA) has been found in the past at both Michigan State and Central Michigan University, and was recently studied in more detail through an NSF planning grant [NSFPGE03]. While the final analysis of the data gathered during the last semester is still not completed, additional evidence was found that such technology helps women to more quickly close the initial gender gap in performance in the sciences. In the science courses we propose to focus on, we traditionally have approximately 50-60\% female students, so we anticipate being able to gather additional statistically significant data. The traditionally relatively low enrollment of ethnic minorities in the courses under consideration might unfortunately not yield statistically significant findings with regards to the improvement of STEM education for underrepresented groups.
+
+\section{Background and Environment}
+\subsection{PI Education and Appointments}
+Dr.~Kortemeyer received his Diplom (ÒM.Sc.Ó) in physics in 1993 from the UniversitŠt Hannover, Germany (Advisor Prof. P. U. Sauer), and his Ph.D. in physics from Michigan State University in 1997 (Advisor Prof. W. Bauer).He has been working at Michigan State University since 1997. His first appointment has been as an Academic Specialist in the Division of Science and Mathematics Education (DSME), where he has been leading instructional technology development projects for the College of Natural Science, and is the director of the Learning{\it Online} Network with Computer-Assisted Personalized Approach (LON-CAPA) project, see section~\ref{loncapa}.  He also taught introductory physics in a completely online mode, as well as co-taught in a more traditional on-campus setting.
 
-If it is then the formulas which are tempting the students to shortcircuit their learning or distract them from it, some physics teachers have turned to almost altogether depriving their students of them. They are developing "conceptual" materials, where {\bf "conceptual" is frequently defined as the absence of formulas}. 
+Starting August 2004, Dr.~Kortemeyer will be working in a tenure-track position as Assistant Professor of Physics Education. His appointment will be split 75/25\% between the Lyman Briggs School of Science (LBS) and DSME. He will also be holding an appointment as Adjunct Professor of Physics in the Department of Physics and Astronomy. His teaching responsibilities will include the introductory calculus-based physics sequence (lecture and lab) in LBS, as well as seminars in special topics. His research will be focused on postsecondary science teaching and learning, with a special emphasis on the use of technology.\subsection{Michigan State University}Michigan State University is one of the earliest land-grant institutions in the United States. MSU is committed to providing equal educational opportunity to all qualified applicants; at the undergraduate level, the university offers comprehensive programs in the liberal arts and sciences, and provides opportunities for students of varying interests, abilities, backgrounds, and expectations.  The total enrollment is approximately 44,000, 35,000 of which are undergraduates. 54\% of the student population are women, 8.1\% African American, 5.1\% Asian/Pacific Islander, 2.8\% Chicano/Other Hispanic, and 0.6\% Native American. Of the freshman class, the average high school GPA is 3.58.\subsection{Lyman Briggs School of Science}The Lyman Briggs School (LBS) at Michigan State University is a residential learning community devoted to studying the natural sciences and their impact on society. All under one roof, LBS encompasses physics, chemistry, biology, and computer laboratories; classrooms; faculty, administrative, and academic support staff offices; student residences; a dining hall; and a convenience store. With approximately 1500 students, LBS offers the benefits of a small, liberal arts college with the resources of a large research university.\subsection{Division of Science and Mathematics Education}The Division of Science and Mathematics Education (DSME) was founded at Michigan State University in 1989, and is co-administered by the College of Natural Science and the College of Education. Academic specialists and faculty members with partial appointments in various departments and other colleges, graduate and undergraduate students, and professional and clerical staff work together in DSME to conduct a variety of research projects, as well as to offer courses, degree programs, and other activities in support of its mission.
+\subsection{Model System: The Learning{\it Online} Network with CAPA}\label{loncapa}For several aspects of the proposed project, the Learning{\it Online} Network with Computer-Assisted Personalized Approach (LON-CAPA; {\tt http://www.lon-capa.org/}) will be the model system. LON-CAPA is an open-source freeware distributed learning content management, course management, and assessment system, and also the model system of the current NSF-ITR grant, see section~\cite{results}. Dr.~Kortemeyer is the director of the LON-CAPA project. \subsubsection{Shared Distributed Content Repository}LON-CAPA is designed around the concept of easy sharing and re-use of learning resources. The ITR research showed that this is only possible by integrating all layers of the infrastructure to give instructors a seamless user experience: pure library systems, such as NSDL, do not offer instructors the ability to in one system:\begin{enumerate}
+\item locate existing content\item generate and publish new content\item enforce secure digital rights management
+\item assemble (sequence) content\item deploy the assembled content in a complete course management system\item have continual assessment of the learning content quality, both subjective and objective\end{enumerate}In addition, the system has to be highly scalable, and avoid single points of failure.In LON-CAPA, the underlying distributed content repository spans all servers in a given cluster. Navigation through selected resources is provided by an internal sequencing tool, which allows assembling, re-using, and re-purposing content at different levels of granularity (pages, lessons, modules, chapters, etc). LON-CAPA provides highly customizable access control on resources, and has a built-in key mechanism to charge for content access. The shared content pool of LON-CAPA currently contains over 60,000 learning resources, including more than 18,000 personalized homework problems. Disciplines include astronomy, biology, business, chemistry, civil engineering, computer science, family and child ecology, geology, human food and nutrition, human medicine, mathematics, medical technology, physics, and psychology. A large fraction of these resources are also available through the gateway to the National Science Digital Library. In addition, the problem supplements to a number of commercial textbooks are available in LON-CAPA format.The network provides constant assessment of the resource quality through objective and subjective dynamic metadata. Selection of a learning resource by instructors at other institutions while constructing a learning module does both establish a de-facto peer-review mechanism and provide additional context information for each resource. In addition, access statistics are being kept, and learners can put evaluation information on each resources.\subsubsection{Formative and Summative Assessment Capabilities}LON-CAPA started in 1992 as a system to give personalized homework to students in introductory physics courses.  ÒPersonalized" means that each student sees a different version of the same computer-generated problem: different numbers, choices, graphs, images, simulation parameters, etc, Fig.~\ref{twoproblems}.
+\begin{figure}\label{twoproblems}
+\includegraphics[width=6.5in]{atwood}
+\caption{Web-rendering of the same LON-CAPA problem for two different students.}
+\end{figure}Students can thus discuss the problems and collaborate on finding solutions, but not simply exchange the answers.
+
+Any problem can be rendered for a variety of purposes: online homework, printed homework, online exams, and printed exams with answer "bubbles."\subsubsection{Course Management}Over the years, the system added a learning content management system and standard course management features, such as communications, gradebook, etc., which are comparable to commercial course management systems, such as BlackBoard, WebCT, or ANGEL. See 
+Refs.~\cite{features,edutools} for an overview of features, and comparisons to other systems.In addition to standard features, the LON-CAPA delivery and course management layer is designed around STEM education, for example:
+\begin{itemize}\item support for mathematical typesetting throughout (\LaTeX\ inside of XML) Ð formulas are rendered on-the-fly, and can be algorithmically modified through the use of variables inside formulas.\item integrated GNUplot support, such that graphs can be rendered on-the-fly, and allowing additional layered labeling of graphs and images\item support for multi-dimensional symbolic math answers\item full support of physical units\end{itemize}
+\subsubsection{Open-Source Freeware}LON-CAPAÕs core development group is located at Michigan State University, and in addition to faculty members, has a staff of three fulltime programmers, two user support staff, one technician, one graduate student, and one project coordinator. LON-CAPA is open-source (GNU General Public License) freeware, there are no licensing costs associated. The LON-CAPA group offers training and support, as well as hosting.
 
-A somewhat extreme example is "Conceptual Physics" by Paul Hewitt~\cite{hewitt}, see Table~\ref{alternatetext}, a more moderate example are the famous {\it Feynman Lectures on Physics}~\cite{feynmanLectures}, which have a much lower "formula density" than most textbooks actually in use in undergraduate teaching (Blatt~\cite{blatt} was chosen as a random example representating the style of most any textbook currently used). 
+\subsection{Collaborative Learning Laboratory}
+The Lyman-Briggs School of Science Collaborative Learning Laboratory, which is expected to be completed in 2005. It is modeled in part after a setup by the North Carolina State University Physics Education R\&D Group~\cite{ncsu}, and offers a space where students can collaborate on homework while their interactions and online transactions are recorded.
+
+In addition to having whiteboards and wireless laptop computers for students to work with in flexible group settings, the facility will have integrated observation equipment to video- and audio-record student interactions. All recorded information is immediately digitized and made available for transcription and analysis using the Transana~\cite{transana} software system.
+\subsection{Courses}
+The project will be carried out  LBS course sequence LBS 271/272 (calculus-based physics I \& II), which have an enrollment of over 200 students each semester. 
+\section{Classes of Online Formative Assessment Problems}
+Redish~\cite{redish} distinguishes eight classes of exam and homework questions, an adapted version of which will form the general classification scheme for Hypothesis 1b:
+\begin{description}
+\item[Multiple-choice and short-answer questions]
+\item[Multiple-choice multiple-response questions]
+\item[Representation-translation questions]
+\item[Ranking-tasks]
+\item[Context-based reasoning problems]
+\item[Estimation problems]
+\item[Qualitative questions]
+\item[Essay questions]
+\end{description}
 
-\begin{table}\label{alternatetext}
-\footnotesize
-\begin{tabular}{p{3in}|p{3in}}
-If $N$ coils of wire, wound in the same sense, are connected, as for example in a solenoid, the induced emf is given by
-\[\varepsilon=-N\frac{\Delta\Phi}{\Delta t}\]
-which is the usual form in which Faraday's law is written.&
-Electromagnetic induction can be summarized in the statement\vspace{2mm}
-
-{\it\bf The induced voltage in a coil is proportional to the product of the number of loops and the rate at which the magnetic field changes within those loops.}\vspace{2mm}
-
-This statement is called {\bf Faraday's Law}.\\\hline
-The resistance of the wire is defined as the ratio of voltage to current; that is,
-\[R=\frac{V}{I}\]
-where $R$ is the resistance, $I$ is the current that flows through this resistance, and $V$ is the {\it potential drop} across the resistance.&
-Ohm discovered that the amount of current in a circuit is directly proportional to the voltage (emf) impressed across the circuit and is inversely proportional to the resistance in of the circuit. In short,
-\[\mbox{Current}=\frac{\mbox{voltage}}{\mbox{resistance}}\]
-Or, in units form,
-\[\mbox{Amperes}=\frac{\mbox{volts}}{\mbox{ohms}}\]
-\end{tabular}
-\caption{Ohm's and Faraday's Law stated in two textbooks,  {\it Principles of Physics} (Blatt\cite{blatt}), left, and {\it Conceptual Physics} (Hewitt\cite{hewitt}), right}
-\end{table}
-
-It should be noted that Hewitt also argues that the mathematical language of physics often deters the average nonscience students~\cite{hewitt}, which is contradictory to the observation of students being very satisfied to it for plug-and-chug.
-\subsection{Intellectual Merit}Much effort and many resources have been invested into developing effective curricular material and assessment, especially in the interactive or online realm, yet very little research has been done on the impact of different representations and question types on student conceptual understanding.The outcomes of this study will provide a broader research base for STEM curriculum development efforts regarding the most effective use of learner feedback. The outcomes will also inform development efforts for online course and learning content management systems, as well as provide input for educational metadata, content exchange, and interoperability standard efforts.\subsection{Broader Impact/Diversity}Attrition in STEM courses is a national problem and may be particularly acute in large, introductory courses that are the staples of many undergraduate science programs, with 40 to 60\% of students leaving STEM disciplines [Committee99, Seymour97]. Much of this attrition has been attributed to inadequate curriculum design, pedagogy and assessment.It is broadly accepted that frequent formative assessment and feedback are a key component of the learning process \cite{bransford}. Shifting the focus from summative to formative assessment with feedback can move student motivation from an extrinsic reward to the intrinsic reward of developing understanding of the materials [Stipek96]. Intrinsic motivation and positive feedback promote the feelings of competence, confidence [Pascarella02, Clark98], and engagement that are crucial to retaining students in introductory STEM courses.  While improving student self-efficacy should have positive impacts on all student retention, Seymour and Hewitt [Seymour97] suggest that such changes should have a particularly strong impact on women and underrepresented groups who may feel that science excludes them. They report that even women with good academic records nonetheless lost confidence in judging their academic performances as "good enough" if they did not receive feedback through personal teacher-student relationships Ð these relationships were reported to be rare, possibly due to the disproportionate number of male teaching staff in STEM courses. Lack of confidence, paired with a perceived gender or race bias, can negatively influence performance [Steele97].A gender-differential benefit of using online formative assessment systems (particularly LON-CAPA) has been found in the past at both Michigan State and Central Michigan University, and was recently studied in more detail through an NSF planning grant [NSFPGE03]. While the final analysis of the data gathered during the last semester is still not completed, additional evidence was found that such technology helps women to more quickly close the initial gender gap in performance in the sciences. In the science courses we propose to focus on, we traditionally have approximately 50-60\% female students, so we anticipate being able to gather additional statistically significant data. The traditionally relatively low enrollment of ethnic minorities in the courses under consideration might unfortunately not yield statistically significant findings with regards to the improvement of STEM education for underrepresented groups.
-\section{Online Formative Assessment}
 \begin{figure}\label{trajectory}
 \includegraphics[width=6.5in]{trajectoryjpg}
 \caption{Web-rendering of the same LON-CAPA problem for two different students.}
@@ -142,39 +154,31 @@
 \includegraphics[width=6.5in]{threemassesjpg}
 \caption{Web-rendering of the same LON-CAPA problem for two different students.}
 \end{figure}
-\section{Hypotheses}
 
+\section{Research Methodology}
+\subsection{Establishment of Initial Conditions}The validity of the hypotheses may depend on both learner and assessment characteristics.\subsubsection{Learner Attitudes, Beliefs, and Expectations}Instruments have been developed to assess epistemological beliefs, for example the Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument~\cite{EBAPS}. Related to epistemological beliefs are learnerÕs expectations and attitudes, and of particular interest is the Maryland Physics Expectations (MPEX) survey~\cite{MPEX}.\subsubsection{Learner Knowledge about the Topic}We will use existing concept inventory surveys as both pre- and post-tests.The qualitative Force Concept Inventory~\cite{fci} and the quantitative companion Mechanical Baseline Test~\cite{hestenesmech} have been used in a large number of studies connected to the teaching of introductory mechanics. The Foundation Coalition has been developing a number of relevant concept inventories~\cite{foundation}, namely the Thermodynamics Concept Inventory, the Dynamics Concept Inventory, and the Electromagnetics Concept Inventory (with two subcomponents, namely Waves and Fields).  Since these were designed from an engineering point of view, some adjustment might be necessary. In addition, the Conceptual Survey of Electricity and Magnetism (CSEM)~\cite{maloney} is available for the second semester course.
 
-\section{Background and Environment}
-\subsection{PI Education and Appointments}
-Dr. Kortemeyer received his Diplom (ÒM.Sc.Ó) in physics in 1993 from the UniversitŠt Hannover, Germany (Advisor Prof. P. U. Sauer), and his Ph.D. in physics from Michigan State University in 1997 (Advisor Prof. W. Bauer).He has been working at Michigan State University since 1997. His first appointment has been as an Academic Specialist in the Division of Science and Mathematics Education (DSME), where he has been leading instructional technology development projects for the College of Natural Science, and is the director of the Learning{\it Online} Network with Computer-Assisted Personalized Approach (LON-CAPA) project, see section~\ref{loncapa}.  He also taught introductory physics in a completely online mode, as well as co-taught in a more traditional on-campus setting.
+4.2.3 Assessment LevelOnline problems will be categorized according to BloomÕs taxonomy [Bloom56]. We will distinguished between ÒLevel 1Ó questions, which only require minimal interpretation of the presented content (corresponding to the Knowledge Level of BloomÕs taxonomy) and ÒLevel 2Ó questions (Comprehension and Application), which require learners to make connections between presented material and apply them to more complex scenarios.4.2.4 Assessment Difficulty and Baseline Statistical DataLON-CAPA automatically keeps tracks of the average number of attempts until a problem is solved, as well as the degree of difficulty and the degree of discrimination. This data is cumulative across semesters, and already exists for all assessment problems from their deployment in previous semesters.4.3 Observables4.3.1 EffectivenessEffectiveness will be measured both in terms of performance on summative assessments (quizzes and exams) and on pre-/post-test concept inventory surveys (section 4.2.2).  Each item on these instruments will be associated with topically corresponding formative online exercises to determine correlations and differential gain between the feedback types used with the respective online problems. A second posttest, correlated with first semester problems, will be administered at the end of the second semester to determine long-term effects.4.3.2 Problem Solving TechniqueWe intend to focus on a subset of students in the LBS Collaborative Learning Laboratory, and observe them while solving problems. Schoenfeld [Schoenfeld85] and Foster [Foster00] developed instruments to categorize and document the stages and expert-like characteristics [Chi81] of observed problem-solving activity by learners, as well as application of metacognitive skills.In addition, we will interview a group of students from all courses regarding their problem-solving strategies. Pascarella [Pascarella02] developed some frameworks for these interviews, which can be built upon.Finally, for all students in all courses, LON-CAPA log data will be analyzed. Kotas [Kotas00] and Minaei [Minaei03] developed a mechanism for this log data analysis, which include submission times between attempts, and quality of the entered input. 4.3.3 Help-Seeking BehaviorIt is impossible to observe all on-demand help seeking, but interactions in several settings can be analyzed:¥ Online discussions and email communication are preserved within LON-CAPA and can be analyzed even in retrospect for past semesters with respect to relevant behavioral patterns. Discussion contributions and states can be linked to online transactions, such as submission of homework attempts, browsing of content material, and hint usage. Wallace [Wallace03] reviewed existing research on such online interactions, however, some adaptation of several of the existing conceptualizations will be necessary to account for the nature of science and in particular physics and biology courses.¥ For the subset of LBS students who come to the LBS Collaborative Learning Laboratory, group discussions can to be documented, and linked to online behavior as absolute timing and learner identify are preserved.¥ Minaei [Minaei03] developed data mining strategies to categorize learner behavior, including navigational patterns between assessment and content material, the use of feedback, and communication functions.¥ Self-reporting can be used for several other help-seeking mechanisms, such as textbook use and peer-interaction ([Riffell03a, Riffell03b]).4.3.4 Cross-Cutting Open-Ended Documentation of Learner PerceptionsWe will interview focus groups of students from every course regarding their experiences and perceived relative helpfulness of the different feedback mechanisms. We will ask them to also reflect on how they perceived these mechanisms to influence their problem solving strategies. In the framework of the current NSF ITR grant [NSFITR00] an instrument was developed to gather learner input on perceived time-on-task and perceived helpfulness and functionality of different aspects of the system.4.4 Data Collection and Comparison GroupsFor the data collection, results would be most meaningful, if we divided the course into groups that experience the same assignment with different feedback mechanisms; indeed, this is what we propose to do in later phases of the study regarding the variation of solely the feedback character. However, when varying the number of available tries and the immediacy of the feedback, we will do so concurrently for all students across different problems Ð otherwise, it would be inevitable that students will complain about unfairness compared to the students with more attempts or immediate feedback. Also students with delayed feedback would most certainly confer with students in the other group.
 
-Starting August 2004, Dr. Kortemeyer will be working in a tenure-track position as Assistant Professor of Physics Education. His appointment will be split 75/25\% between the Lyman Briggs School of Science (LBS) and DSME. He will also be holding an appointment as Adjunct Professor of Physics in the Department of Physics and Astronomy. His teaching responsibilities will include the introductory calculus-based physics sequence (lecture and lab) in LBS, as well as seminars in special topics. His research will be focused on postsecondary science teaching and learning, with a special emphasis on the use of technology.\subsection{Michigan State University}Michigan State University is one of the earliest land-grant institutions in the United States. MSU is committed to providing equal educational opportunity to all qualified applicants; at the undergraduate level, the university offers comprehensive programs in the liberal arts and sciences, and provides opportunities for students of varying interests, abilities, backgrounds, and expectations.  The total enrollment is approximately 44,000, 35,000 of which are undergraduates. 54 percent of the student population are women, 8.1 percent African American, 5.1 percent Asian/Pacific Islander, 2.8 percent Chicano/Other Hispanic, and 0.6 percent Native American. Of the freshman class, the average high school GPA is 3.58.\subsection{Lyman Briggs School of Science}The Lyman Briggs School (LBS) at Michigan State University is a residential learning community devoted to studying the natural sciences and their impact on society. All under one roof, LBS encompasses physics, chemistry, biology, and computer laboratories; classrooms; faculty, administrative, and academic support staff offices; student residences; a dining hall; and a convenience store. With approximately 1500 students, LBS offers the benefits of a small, liberal arts college with the resources of a large research university.\subsection{Division of Science and Mathematics Education}The Division of Science and Mathematics Education (DSME) was founded at Michigan State University in 1989, and is co-administered by the College of Natural Science and the College of Education. Academic specialists and faculty members with partial appointments in various departments and other colleges, graduate and undergraduate students, and professional and clerical staff work together in DSME to conduct a variety of research projects, as well as to offer courses, degree programs, and other activities in support of its mission.
-\subsection{Model System: The LearningOnline Network with CAPA}\label{loncapa}For several of the proposed project, the LearningOnline Network with Computer-Assisted Personalized Approach (LON-CAPA) will be the model system. Dr. Kortemeyer is the director of the LON-CAPA project. LON-CAPA is an open-source freeware distributed learning content management, course management, and assessment system. LON-CAPA is the model system of the current NSF-ITR grant Investigation of a Model for Online Resource Creation and Sharing in Educational Settings (\#0085921, \$2,055,000, September 15, 2000 through July 31, 2005).\subsubsection{Shared Distributed Content Repository}Within the ITR project, LON-CAPA evolved into a mature system, which spans 35 institutions of secondary and postsecondary education, serves over 23,000 learners every semester, and holds over 60,000 online learning resources, mostly in the STEM areas.LON-CAPA is designed around the concept of easy sharing and re-use of learning resources. The ITR research showed that this is only possible by integrating all layers of the infrastructure to give instructors a seamless user experience: pure library systems, such as NSDL, do not offer instructors the ability to in one system:\begin{enumerate}
-\item locate existing content\item generate and publish new content\item enforce secure digital rights management
-\item assemble (sequence) content\item deploy the assembled content in a complete course management system\item have continual assessment of the learning content quality, both subjective and objective\end{enumerate}In addition, the system has to be highly scalable, and avoid single points of failure.In LON-CAPA, the underlying distributed content repository spans all servers in a given cluster. Navigation through selected resources is provided by an internal sequencing tool, which allows assembling, re-using, and re-purposing content at different levels of granularity (pages, lessons, modules, chapters, etc). LON-CAPA provides highly customizable access control on resources, and has a built-in key mechanism to charge for content access. The shared content pool of LON-CAPA currently contains over 60,000 learning resources, including more than 18,000 personalized homework problems.The network provides constant assessment of the resource quality through objective and subjective dynamic metadata. Selection of a learning resource by instructors at other institutions while constructing a learning module does both establish a de-facto peer-review mechanism and provide additional context information for each resource. In addition, access statistics are being kept, and learners can put evaluation information on each resources. A large fraction of these resources are also available through the gateway to the National Science Digital Library. In addition, the problem supplements to a number of commercial textbooks are available in LON-CAPA format.Disciplines include astronomy, biology, business, chemistry, civil engineering, computer science, family and child ecology, geology, human food and nutrition, human medicine, mathematics, medical technology, physics, and psychology.LON-CAPA supports multilingual content resources, and its user interface has been localized in Arabic, Farsi, French, German, Japanese, Portuguese, Russian, and Turkish. \subsubsection{Formative and Summative Assessment}LON-CAPA started in 1992 as a system to give personalized homework to students in introductory physics courses.  ÒPersonalized" means that each student sees a slightly different computer-generated problem: different numbers in numerical problems, different choices in multiple choice-type problems, different graphs, images, simulation parameters, etc, Fig.~\ref{twoproblems}.\begin{figure}\label{twoproblems}
-\includegraphics[width=6.5in]{atwood}
-\caption{Web-rendering of the same LON-CAPA problem for two different students.}
-\end{figure}Students can thus discuss the problems and collaborate on finding solutions, but not simply exchange the answers. 
-\subsubsection{Course Management}Over the years, the system added a learning content management system and standard course management features, such as communications, gradebook, etc., which are comparable to commercial course management systems, such as BlackBoard, WebCT, or ANGEL. See http://www.lon-capa.org/features.html and http://www.edutools.info/course/compare/ for an overview of features, and comparisons to other systems.In addition to standard features, the LON-CAPA delivery and course management layer is designed around STEM education, for example:
-\begin{itemize}\item support for mathematical typesetting throughout (\LaTeX\ inside of XML) Ð formulas are rendered on-the-fly, and can be algorithmically modified through the use of variables inside formulas.\item integrated GNUplot support, such that graphs can be rendered on-the-fly, and allowing additional layered labeling of graphs and images\item support for multi-dimensional symbolic math answers\item full support of physical units\end{itemize}
-\subsubsection{Open-Source Freeware}LON-CAPAÕs core development group is located at Michigan State University, and in addition to faculty members, has a staff of three fulltime programmers, two user support staff, one technician, one graduate student, and one project coordinator. LON-CAPA is open-source (GNU General Public License) freeware, there are no licensing costs associated. The LON-CAPA group offers training and support, as well as hosting.
+\section{Evaluation}
+The LON-CAPA Faculty Advisory Board was formed as part of our NSF ITR grant project. It consists of eight actively teaching faculty and administrators from a number of colleges on campus of MSU, and meets once every month to both evaluate and advise projects connected to LON-CAPA. We propose to continue using this existing structure to evaluate this projectÕs progress and findings. In addition, Dr.~Kortemeyer's Mentoring Committee, which consists of senior faculty members from both LBS and DSME will guide and advise the progress of this project.
+\section{Dissemination}
+We will present papers at conferences such as the LON-CAPA User Conference, Frontiers in Education, and the American Association of Physics Teachers Annual Meeting.  We will submit papers to journals such as The Physics Teacher, the American Journal of Physics, Computers and Education, and the Journal of Asynchronous Learning Networks.  Finally, any content material adapted and implemented in this project will be immediately available to all participant LON-CAPA institutions and via the LON-CAPA gateway to the NSF-funded National Science Digital Library.
+\section{Project Timeline}
 
 
-\begin{figure}\label{conceptproblems}
-\includegraphics[width=6.5in]{threemassesjpg}
-\caption{Traditional and "conceptual" LON-CAPA physics problem.}
-\end{figure}
-
 
-\section{Results from Prior NSF Support}
-Gerd Kortemeyer is PI on the current NSF-ITR grant Investigation of a Model for Online Resource Creation and Sharing in Educational Settings (\#0085921, \$2,055,000, September 15, 2000 through July 31, 2005), which uses LON-CAPA as its model system. The project is designed to address questions of resource pooling and sharing across content areas. The investigators are incubating a multi-institutional collaboration and bring together stakeholders to address content issues such as reuse, customization, online community building, quality, and effectiveness. The project currently has more than 30 participating institutions, and continues to study the formation of its online collaborative community, including workshops, conferences, support, evaluation, and dissemination. The project maintains a gateway server to the National STEM Digital Library, and the LON-CAPA shared resource pool is searchable and accessible from http://nsdl.org/. Gerd Kortemeyer is Co-PIs on the current NSF-CCLI-ASA grant Diagnostic Question Clusters: Development and Testing in Introductory Geology and Biology (\#0243126, \$491,606, September 15, 2003 through August 31, 2006) to develop diagnostic questions for college students in both biology and geology. In the ASA project, a pool of peer-reviewed, diagnostic question clusters to assess students' understanding will be developed, including tools for analysis, peer review, and online publication of these question clusters.
+\section{Results from Prior NSF Support}\label{results}
+Gerd Kortemeyer is PI on the current NSF-ITR grant Investigation of a Model for Online Resource Creation and Sharing in Educational Settings (\#0085921, \$2,055,000, September 15, 2000 through July 31, 2005), which uses LON-CAPA as its model system. The project is designed to address questions of resource pooling and sharing across content areas. The investigators are incubating a multi-institutional collaboration and bring together stakeholders to address content issues such as reuse, customization, online community building, quality, and effectiveness. The project currently has more than 30 participating institutions, and continues to study the formation of its online collaborative community, including workshops, conferences, support, evaluation, and dissemination. The project maintains a gateway server to the National Science Digital Library, and the LON-CAPA shared resource pool is searchable and accessible from {\tt http://nsdl.org/}. Gerd Kortemeyer is Co-PIs on the current NSF-CCLI-ASA grant Diagnostic Question Clusters: Development and Testing in Introductory Geology and Biology (\#0243126, \$491,606, September 15, 2003 through August 31, 2006) to develop diagnostic questions for college students in both biology and geology. In the ASA project, a pool of peer-reviewed, diagnostic question clusters to assess students' understanding will be developed, including tools for analysis, peer review, and online publication of these question clusters.
 
 % references
 \newpage
 
 \pagestyle{plain}
 \begin{thebibliography}{99}
+
+% Intro
+
 \bibitem{feynmanCharacter} Richard Feynman, {\it The Character of Physical Law}, The MIT Press, ISBN 0 262 56003 8 (1967)
 \bibitem{student} Student online discussion contribution, "Introductory Physics for Scientists and Engineers," phy183, Michigan State University (2004)
 \bibitem{heuvelen} Alan Van Heuvelen, {\it Learning to think like a physicist: A review of research-based instructional strategies}, Am. J. Phys. {\bf 59}(10), 891-897 (1991)
@@ -194,10 +198,30 @@
 \bibitem{thoennessen} M. Thoennessen and M. J. Harrison, {\it Computer-Assisted Assignments in a Large Physics Class}, Comp. Educ. {\bf 27}, 141 (1996)
 \bibitem{kashy00} E. Kashy, M. Thoennessen Y. Tsai, N. E. Davis, and G. Albertelli II, {\it Melding Network Technology with Traditional Teaching: Enhanced Achievement in a 500-Student Course}, Chapter in {\it Interactive Learning: Vignettes from America's Most Wired Campuses}, 
 David G. Brown (editor), Anker Publishing Company, Boston, 51, ISBN 1-882982-29-0 (2000)
+\bibitem{hewitt} Paul G. Hewitt, {\it Conceptual Physics}, Little, Brown, ISBN 0-673-39541-3
+\bibitem{stipek} D. J. Stipek, {\it Motivation and instruction}, in D.C. Berliner and R.C. Calfee (Eds.), {\it Handbook of educational psychology}, New York:  Macmillan Library, 85-113 (1996)\bibitem{clark} K. Clark and F. Dwyer, {\it Effect of different types of computer-assisted feedback strategies on achievement and response confidence}, International Journal of Instructional Media {\bf 25}(1), 55-63 (1998)
+\bibitem{pascarella02} A. M. Pascarella, {\it CAPA (Computer-Assisted Personalized Assignments) in a Large University Setting}, Ph.D. (Physics) dissertation, University of Colorado (2002)
+
+% Background
+
+\bibitem{features} LON-CAPA feature overview, 
+{\tt http://www.lon-capa.org/features.html}
+\bibitem{edutools} EDUTOOLS Course Management System Decision Making Tools, {\tt http://www.edutools.info/course/compare/} 
+\bibitem{ncsu} North Carolina State University, {\tt http://www.physics.ncsu.edu:8380/physics\_ed/Room\_Design\_files/frame.htm}
+\bibitem{transana} University of Wisconsin, Wisconsin Center for Education Research, {\tt http://www2.wcer.wisc.edu/Transana/}
+
+% Methodology
+\bibitem{EBAPS} A. Elby, J. Fredriksen, C. Schwarz, and B. White, {\it Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument}, {\tt http://www2.physics.umd.edu/$\sim$elby/EBAPS/EBAPS\_items.htm} (2001)
+\bibitem{MPEX} E. F. Redish, J. M. Saul, and R. N. Steinberg, {\it Student expectations in introductory physics}, Am. J. Physics {\bf 66}, 212-224 (1998), survey available on The Physics Suite CD in Teaching Physics, ISBN 0-471-39378-9 (2003)
+\bibitem{hestenesmech} D. Hestenes and M.Wells, {\it A Mechanics Baseline Test}, Phys. Teach {\bf 30}(3), 159-166 (1992)
+\bibitem{foundation} Foundation Coalition, {\it Key Components: Concept Inventories}, {\tt http://www.foundationcoalition.org/home/keycomponents/concept/index.html} (2003)
+\bibitem{maloney} D. P. Maloney, T. L. OÕKuma, C. J. Hieggelke, A. van Heuvelen, {\it Surveying studentsÕ conceptual knowledge of electricity and magnetism}, Am. J. Phys. Suppl. {\bf 69}, S12 (2001)
+
+% ---- UNUSED
 
+\bibitem{torigoe} Eugene Torigoe, {\it Student Difficulties with Equations in Physics}, ISAAPT Spring Meeting, Urbana, IL, (April 2004)
+\bibitem{clement} J. Clement, J. Lochhead, and G. S. Monk, {\it Translation difficulties in learning mathematics}, Amer. Math. Mon. {\bf 88}, 286 (1981)
 
-\bibitem{torigoe} Eugene Torigoe, {\it Student Difficulties with Equations in Physics}, ISAAPT Spring Meeting, Urbana, IL, (April 2004)\bibitem{clement} J. Clement, J. Lochhead, and G. S. Monk, {\it Translation difficulties in learning mathematics}, Amer. Math. Mon. {\bf 88}, 286 (1981)
-\bibitem{hewitt} Paul G. Hewitt, {\it Conceptual Physics}, Little, Brown, ISBN 0 673 39541 3
 \bibitem{feynmanLectures} Richard Feynman, Robert B. Leighton, and Matthew L. Sands, {\it The Feynman Lectures on Physics}, Addison-Wesley, ISBN 0-201-5100(3,4,5)-(0,9,0) (1963-65)
 \bibitem{blatt} Frank J. Blatt, {\it Principles of Physics}, Allyn and Bacon, ISBN 0 205 11784 8
 

--www1089300964--