[LON-CAPA-cvs] cvs: modules /gerd/concept description.tex
www
lon-capa-cvs@mail.lon-capa.org
Sun, 18 Jul 2004 19:25:15 -0000
This is a MIME encoded message
--www1090178715
Content-Type: text/plain
www Sun Jul 18 15:25:15 2004 EDT
Modified files:
/modules/gerd/concept description.tex
Log:
Hopefully, this is it.
--www1090178715
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20040718152515.txt"
Index: modules/gerd/concept/description.tex
diff -u modules/gerd/concept/description.tex:1.18 modules/gerd/concept/description.tex:1.19
--- modules/gerd/concept/description.tex:1.18 Fri Jul 16 16:48:34 2004
+++ modules/gerd/concept/description.tex Sun Jul 18 15:25:14 2004
@@ -32,7 +32,7 @@
\DeclareGraphicsExtensions{.eps, .jpg}
\fi
\begin{center}
-\LARGE\sc Physics education:\\ Does "conceptual" online formative assessment lead to conceptual understanding?
+\LARGE\sc Physics education:\\ Does ``conceptual" online formative assessment lead to conceptual understanding?
\end{center}
\section{Goals and Objectives}\label{intro}
\begin{quote}
@@ -42,7 +42,7 @@
\begin{quote}
When you guys help us with problems [...] in
-class you really got to start plugging in numbers into the
+class you really [have] to start plugging in numbers into the
equations that you use and solving the problem step by step.
And then at the end coming up with a concrete answer like 2
or 4.5654 (not $y_{0x}$ or $x_o$) so when we look at our notes even
@@ -52,7 +52,7 @@
\end{quote}
\subsection{Overview}
-This five-year project focuses on online formative assessment in introductory physics education, and how it can be used to help learners re-evaluate their epistemologies, develop expertlike problem solving skills, and gain a conceptual understanding of physics. It will compare the impact of online problems which are categorized across 21 types previously identified in literature.
+This five-year project focuses on online formative assessment in introductory physics education, and how it can be used to help learners re-evaluate their epistemologies, develop expertlike problem solving skills, and gain a conceptual understanding of physics. It will compare the effect of different classes of online problems, which are categorized across 21 types previously identified in literature.
The study will be carried out in on-campus courses with regular classroom times, which are enhanced by online components. The NSF-supported Learning{\it Online} Network with Computer-Assisted Personalized Approach (LON-CAPA; Sect.~\ref{loncapa}) will be used as the model system.
@@ -67,20 +67,20 @@
\item hypotheses-testing for each problem type regarding its educational impact (Sect.~\ref{analysis})
\end{enumerate}
-\subsection{"Thinking like a Physicist"}
-Most physicists would agree that part of "being a physicist" is to "think like a physicist"~\cite{heuvelen}, which manifests itself most prominently in the cognitive and metacognitive skills involved in problem solving~\cite{redish} --- part of the self-image of many physicists is to be expert problem solvers, even outside their own discipline~\cite{fuller}:
+\subsection{``Thinking like a Physicist"}
+Most physicists would agree that part of ``being a physicist" is ``thinking like a physicist"~\cite{heuvelen}, which manifests itself most prominently in the cognitive and metacognitive skills involved in problem solving~\cite{redish} --- part of the self-image of many physicists is to be expert problem solvers, even outside of their own discipline~\cite{fuller}:
\[\mbox{Think like a Physicist}\ \Rightarrow\ \mbox{Solve Problems}\]
Yet, while basic logic tells us that the reverse statement, i.e.,
\[\mbox{Think like a Physicist}\ \Leftarrow\ \mbox{Solve Problems}\]
does not necessarily have to be true,
an unfortunately all too common approach to teaching physics is to simply have students solve a lot of standard problems.
-It is broadly accepted that frequent formative assessment and feedback are a key component of the learning process~\cite{bransford}, but {\it what} the students are learning is not necessarily what educators are expecting them to be learning~\cite{pellegrino,arons}. This disconnect frequently goes undetected if the summative assessment tools are similar to the formative assessment tools~\cite{lin}. In most physics courses, these are conventional problems along the lines of "A ball starts with an initial velocity of \ldots. \ldots What is the \ldots?". Deploying alternative conceptual assessment tools such as the Force Concept Inventory~\cite{fci} can reveal large discrepancies between different summative assessment types (e.g,~\cite{steinberg,mazur}).
+It is broadly accepted that frequent formative assessment and feedback are a key component of the learning process~\cite{bransford}, but {\it what} the students are learning is not necessarily what educators are expecting them to be learning~\cite{pellegrino,arons}. This disconnect frequently goes undetected if the summative assessment tools are similar to the formative assessment tools~\cite{lin}. In most physics courses, these are conventional problems along the lines of ``A ball starts with an initial velocity of \ldots. \ldots What is the \ldots?". Deploying alternative conceptual assessment tools such as the Force Concept Inventory~\cite{fci} can reveal large discrepancies between different summative assessment types (e.g,~\cite{steinberg,mazur}).
Expert and novice approaches to problem solving in physics have been studied extensively (e.g.~\cite{chi,larkin}). Two of the most apparent differences are that
\begin{enumerate}
-\item experts are initially characterizing problems according to deep structure and physical concepts (e.g., "energy conservation"-problem), while novices tend to characterize them according to surface features (e.g., "sliding-block-on-incline"-problem) or applicable formulas (e.g., "$E=\frac12mv^2+mgh$"-problem")
-\item novices then continue to employ a formula-centered problem solving method~\cite{heuvelen}, frequently referred to as "plug-and-chug."
+\item experts initially characterize problems according to deep structure and physical concepts (e.g., "energy conservation"-problem), while novices tend to characterize them according to surface features (e.g., ``sliding-block-on-incline"-problem) or applicable formulas (e.g., ``$E=\frac12mv^2+mgh$"-problem")
+\item novices then continue to employ a formula-centered problem solving method~\cite{heuvelen}, frequently referred to as ``plug-and-chug."
\end{enumerate}
Redish~\cite{redish} somewhat bleakly describes a novice approach to learning physics as follows:
\begin{itemize}
@@ -91,47 +91,47 @@
\item Erase all information from your brain after the exam to make room for the next set of materials.
\end{itemize}
-One cannot really blame learners for short-circuiting physics "learning" this way, since the cognitive and metacognitive skills, which physicists value so highly, are hardly ever made explicit, neither in instruction, nor in formative or summative assessment~\cite{lin,reif,mazur96}; in fact, they are mostly altogether "hidden"~\cite{redish} from all aspects of a course, and students are affirmed in their novice expectations~\cite{hammer} of what it is to "do physics." The challenge is to move students away from treating physics as a set of unrelated factoids and formulas, as well as away from focussing on memorizing and using formulas without interpretation or sense-making~\cite{hammer}, and toward both "thinking like a physicist" and gaining conceptual understanding.
+One cannot really blame learners for short-circuiting physics "learning" this way, since the cognitive and metacognitive skills, which physicists value so highly, are hardly ever made explicit, neither in instruction, nor in formative or summative assessment~\cite{lin,reif,mazur96}; in fact, they are mostly altogether ``hidden"~\cite{redish} from all aspects of a course, and students are affirmed in their novice expectations~\cite{hammer} of what it is to ``do physics." The challenge is to move students away from treating physics as a set of unrelated factoids and formulas, as well as away from focusing on memorizing and using formulas without interpretation or sense-making~\cite{hammer}, and toward both ``thinking like a physicist" and gaining conceptual understanding.
-"Conceptual understanding" in this project is defined as insight,
+As a result, some instructors turn to ``conceptual" curricular material, where however ``conceptual" often seems to be defined simply as ``lacking numbers and formulas." The term ``conceptual" will need a more specific definition if it is meant to indeed denote ``fostering conceptual understanding." ``Conceptual understanding" in this project is defined as insight,
as reflected in thoughtful and effective use of knowledge and skills in varied situations,
into abstract key ideas,
which are generalized from particular instances.
\subsection{The Problem with Problems - Hypotheses}\label{hypo}
-To quote Lin~\cite{lin}: "The primary determinants of student performance are the specific tasks for which teachers explicitly hold students responsible (e.g. problem sets and exams), rather than the general goals of the teacher (e.g. conveying an appreciation of the power of physics in a broad context)." Mazur~\cite{mazur96} asks "So why do we keep testing our students with conventional problems?"
+To quote Lin~\cite{lin}: ``The primary determinants of student performance are the specific tasks for which teachers explicitly hold students responsible (e.g. problem sets and exams), rather than the general goals of the teacher (e.g. conveying an appreciation of the power of physics in a broad context)." Mazur~\cite{mazur96} asks ``So why do we keep testing our students with conventional problems?"
The answer, only too often, is scalability, and that in more than one dimension: non-conventional problems are harder to write, and even harder to grade.
The scalability problem is easier to overcome in the classroom: alternative formative assessment as a classroom tool, where students are forced to verbally express their views and teach each other, rather than calculate answers~\cite{mazur}, is starting to be adopted as an effective teaching practice in more and more courses.
-It would clearly be advantageous to extend these effective verbalization practices outside the classroom, and offer formative assessment opportunities in which students get to work through and write about real-life problems on a conceptual level, and are explicitly graded on formulating assumptions, developing models, doing back-of-the-envelope estimations, and deriving relevant formulas and solutions. Given both time and logistical constraints, except for the occasional "project assignment," that is not a reality.
+It would clearly be advantageous to extend these effective verbalization practices outside the classroom, and offer formative assessment opportunities in which students get to work through and write about real-life problems on a conceptual level, and are explicitly graded on formulating assumptions, developing models, doing back-of-the-envelope estimations, and deriving relevant formulas and solutions. Given both time and logistical constraints, except for the occasional ``project assignment," that is not a reality.
-This project focusses on how to move beyond conventional homework problems while operating within the realistic limitations of large-enrollment courses.
+This project focuses on how to move beyond conventional homework problems while operating within the realistic limitations of large-enrollment courses.
Particularly in large-enrollment courses, timely feedback is often impossible without the use of computerized homework systems (e.g.~\cite{thoennessen,kashy00}). Unfortunately, an all too frequent approach to using such systems is to simply replicate conventional textbook problems in the online realm, where they are conveniently graded by the computer.
-The project assumes that "the problem with problems" (a phrase borrowed from~\cite{mazur96}) is that
+The project assumes that ``the problem with problems" (a phrase borrowed from~\cite{mazur96}) is that
\begin{quote}
{\bf Hypothesis 1a:} Conventional calculation-oriented problems affirm non-expertlike epistemologies and encourage non-expertlike problem-solving strategies
\end{quote}
Using computerized systems does impose limitations on which kind of problems can be made available, but does not limit educators to just these most basic types. A further assumption is that by the reverse token
\begin{quote}
-{\bf Hypothesis 1b:} There are types of online formative assessment computer-evaluated problems which make learners confront their non-expertlike epistemologies and encourage expertlike problem-solving strategies\end{quote}
+{\bf Hypothesis 1b:} There are types of online (mostly) computer-evaluated problems, which make learners confront their non-expertlike epistemologies and encourage expertlike problem-solving strategies\end{quote}
These problem types might involve both computer- and human-evaluated components, where an emphasis has to be put on keeping the human-evaluated part manageable and scalable.
-An additional problem with conventional problems may be their mathematical nature. Hewitt in the preface to his textbook "Conceptual Physics"~\cite{hewitt} argues that the mathematical language of physics often deters the average non-science students, a notion which concurs with Tobias' concept of "math anxiety"\cite{tobias}, which is a particular issue for students in the "second tier"\cite{tobiasST} of science courses. For them, the use of mathematics in physics courses can present a hurdle, and a lack of skills or confidence to perform basic algebraic manipulations ("$V=RI\ \Rightarrow\ R=V/I"$), or even problems operating their pocket calculators, can hinder students' learning progress in physics at a very basic level.
+An additional problem with conventional problems may be their mathematical nature. Hewitt in the preface to his textbook ``Conceptual Physics"~\cite{hewitt} argues that the mathematical language of physics often deters the average non-science students, a notion which concurs with Tobias' concept of ``math anxiety"\cite{tobias}, which is a particular issue for students in the ``second tier"\cite{tobiasST} of science courses. For them, the use of mathematics in physics courses can present a hurdle, and a lack of skills or confidence to perform basic algebraic manipulations (``$V=RI\ \Rightarrow\ R=V/I"$), or even problems operating their pocket calculators, can hinder students' learning progress in physics at a very basic level.
-Yet, the majority of students appears to be able to correctly substitute variables and execute calculations, and is quite content with the "plug-and-chug" approach. In fact, it appears to be true that their "concept anxiety" is more prominent than any "math anxiety."
+Yet, the majority of students appears to be able to correctly substitute variables and execute calculations, and is quite content with the ``plug-and-chug" approach. In fact, it appears to be true that their ``concept anxiety" is more prominent than any ``math anxiety."
Moving beyond initial barriers, the problem with mathematics as part of a formative assessment appears to be not one of {\it operation}, but one of {\it translation}. Students see formulas in a purely operational sense~\cite{torigoe,breitenberger}, while lacking the ability to translate between the formulas and the situations~\cite{clement}, which is also illustrated in the expert and novice quotes at the beginning of Sect.~\ref{intro}.
-Online homework systems by the very nature of computers lend themselves to standard calcu\-lation-oriented problems, and are extensively used in this way. Yet, the "plug-and-chug" approach is the most prominent symptom of novice-like problem-solving strategy, and calculation-oriented problems may encourage just that. As a result, there is a frequent call for "conceptual" online problems, where both instructors and students seem to define "conceptual" simply by the absence of numbers and formulas.
+Online homework systems by the very nature of computers lend themselves to standard calcu\-lation-oriented problems, and are extensively used in this way. Yet, the ``plug-and-chug" approach is the most prominent symptom of novice-like problem-solving strategy, and calculation-oriented problems may encourage just that. As a result, there is a frequent call for ``conceptual" online problems, where both instructors and students seem to define ``conceptual" simply by the absence of numbers and formulas.
\begin{itemize}
-\item But does "depriving" students of numbers and formulas indeed make them work on a conceptual level?
-\item Does it help both students who have problems with applying mathematical methods and those who comfortably "plug-and-chug" gain conceptual understanding of physics?
+\item But does ``depriving" students of numbers and formulas indeed make them work on a conceptual level?
+\item Does it help both students who have problems with applying mathematical methods and those who comfortably ``plug-and-chug" gain conceptual understanding of physics?
\end{itemize}
The following hypotheses reflect these notions in the positive form:
\begin{quote}
@@ -140,13 +140,13 @@
\begin{quote}
{\bf Hypothesis 3:} Learners with an average or above average level of mathematical skills or confidence will more likely develop a conceptual understanding of physics as a result of non-calculation-oriented online formative assessment, by discouraging non-expertlike problem-solving strategies.
\end{quote}
-As evidenced in the above definition, it should be emphasized that the project does by no means attempt to establish or promote a dichotomy between "conceptual understanding" and "basic skills/factual knowledge." A physicist needs basic skills and factual knowledge, and the learning of these must not be underemphasized in formative assessment. However, how to best develop these through formative assessment would constitute another valid research project.
+As evidenced in the above definition, it should be emphasized that the project does by no means attempt to establish or promote a dichotomy between ``conceptual understanding" and ``basic skills/factual knowledge." A physicist needs basic skills and factual knowledge, and the learning of these must not be underemphasized in formative assessment. However, how to best develop these through formative assessment would constitute another valid research project.
\subsection{Intellectual Merit}
-Online homework is becoming increasingly prominent in physics education, yet research into its effect has been contradictory, sparse, and, in some cases, not very systematic~\cite{pasc04}. Differences have been reported positive~\cite{kashyda}, negative~\cite{pasc04}, and non-significant~\cite{bonham}. Pascarella~\cite{pascarella02} was one of the very few studies investigating problem-solving strategies, but both Pascarella~\cite{pascarella02} and Bonham~\cite{bonham} only considered the online versions of conventional textbook-like problems. Kashy~\cite{kashyd01b} found that what the authors call "interactive problems," namely those where learners need to read relevant values from graphs or observe simulations, are better predictors of overall success in the course than other problem types, but did not investigate cause and effect relationships, or study problem-solving behavior.
+Online homework is becoming increasingly prominent in physics education, yet research into its effect has been contradictory, sparse, and, in some cases, not very systematic~\cite{pasc04}. Differences have been reported positive~\cite{kashyda}, negative~\cite{pasc04}, and non-significant~\cite{bonham}. Pascarella~\cite{pascarella02} was one of the very few studies investigating problem-solving strategies, but both Pascarella~\cite{pascarella02} and Bonham~\cite{bonham} only considered the online versions of conventional textbook-like problems. Kashy~\cite{kashyd01b} found that what the authors call ``interactive problems," namely those where learners need to read relevant values from graphs or observe simulations, are better predictors of overall success in the course than other problem types, but did not investigate cause and effect relationships, or study problem-solving behavior.
This study aims to provide a systematic research base regarding the effectiveness of different types of online formative assessment, especially those which do take better advantage of the medium, and inform both curriculum development efforts and practitioners.
\subsection{Broader Impact/Diversity}
-Currently, every semester approximately 350,000 students are taking introductory undergraduate physics courses similar to the ones under investigation in this project~\cite{aapt}. For many of these students, it is both their first and their last formal exposure to physics. Students will go into a large spectrum of careers, with or without an understanding of the basic concepts of the physical world.
+Currently, every semester approximately 350,000 US students are taking introductory undergraduate physics courses similar to the ones under investigation in this project~\cite{aapt}. For many of these students, it is both their first and their last formal exposure to physics. Students will go into a large spectrum of careers, with or without an understanding of the basic concepts of the physical world.
This project has the potential of broader impact, since like many of the other efforts in Physics Education, it is carried out within a regular college venue. Results from this study will be applicable especially in large-enrollment courses, where for logistical reasons online homework is frequently the only feasible formative assessment mechanism. Both the tool (LON-CAPA, Sect.~\ref{loncapa}) and any developed, implemented, and adapted materials (Sect.~\ref{matdev}) will be readily available to physics faculty. Faculty members at the over thirty currently participating LON-CAPA institutions will be able to profit from this project already during its progress.
@@ -174,7 +174,7 @@
In LON-CAPA, the underlying distributed multimedia content repository spans across all of the currently over 30 participating institutions, and currently contains over 60,000 learning content resources, including more than 18,000 personalized homework problems. Disciplines include astronomy, biology, business, chemistry, civil engineering, computer science, family and child ecology, geology, human food and nutrition, human medicine, mathematics, medical technology, physics, and psychology. Any content material contributed to the pool is immediately available and ready-to-use within the system at all participating sites, thus facilitating dissemination of curricular development efforts (Sect.~\ref{matdev}). A large fraction of these resources are also available through the gateway to the National Science Digital Library (NSDL).
Navigation through selected resources is provided by an internal sequencing tool, which allows assembling, re-using, and re-purposing content at different levels of granularity (pages, lessons, modules, chapters, etc) --- each content assembly becomes a new resource in the system.
-The network provides constant assessment of the resource quality through objective and subjective dynamic metadata. Selection of a learning resource by instructors at other institutions while constructing a learning module does both establish a de-facto peer-review mechanism and provide additional context information for each resource. In addition, access statistics are being kept, and learners can put evaluation information on each resources.
+The network provides constant assessment of the resource quality through objective and subjective dynamic metadata. Selection of a learning resource by instructors at other institutions while constructing a learning module does both establish a de-facto peer-review mechanism and provide additional context information for each resource. In addition, access statistics are being kept, and learners can put evaluation information on each resource.
In addition to faculty-provided content, the problem supplements to a number of commercial textbooks are available in LON-CAPA format.
LON-CAPA provides highly customizable access control for such resources, and has a built-in key mechanism to charge for content access. \subsubsection{Formative and Summative Assessment Capabilities}LON-CAPA started in 1992 as a system to give personalized homework to students in introductory physics courses. ÒPersonalized" means that each student sees a different version of the same computer-generated problem: different numbers, choices, graphs, images, simulation parameters, etc, Fig.~\ref{twoproblems}.
@@ -212,7 +212,7 @@
\end{figure}
\subsection{Collaborative Learning Laboratory}
-The LBS Collaborative Learning Laboratory, which is expected to be completed in 2005. It is modeled in part after a setup by the North Carolina State University Physics Education R\&D Group~\cite{ncsu}, and offers a space where students can collaborate on homework while their interactions and online transactions are recorded.
+The LBS Collaborative Learning Laboratory is expected to be completed in 2005. It is modeled in part after a setup by the North Carolina State University Physics Education R\&D Group~\cite{ncsu}, and offers a space where students can collaborate on homework while their interactions and online transactions are recorded.
In addition to having whiteboards and wireless laptop computers for students to work with in flexible group settings, the facility will have integrated observation equipment to video- and audio-record student interactions. All recorded information is immediately digitized and made available for transcription and analysis using the Transana~\cite{transana} software system.
@@ -226,39 +226,39 @@
Students in these courses are currently solving approximately 200 online homework problems each semester, most of which currently are of the conventional type.
\section{Classification of Online Formative Assessment Problems}\label{class}
-Redish~\cite{redish} distinguishes eight types of exam and homework questions, an adapted version of which will form the general classification scheme (Table~\ref{classification}) for Hypothesis 1b:
+Redish~\cite{redish} identifies eight classes and features of exam and homework questions, an adapted version of which will form the general classification scheme (Table~\ref{classification}) for Hypothesis 1b:
\begin{description}
\item[Multiple-choice and short-answer questions] The most basic and most easily computer-evaluated type of question, representing the conventional (typical back-of-chapter textbook) problem.
-For the purposes of this project, "multiple choice" and "short-answer" will be considered as separate classes, where short-answer includes numerical answers such as "$17 kg/m^3$," and formula answers, such as "\verb!1/2*m*(vx^2+vy^2)!." The problems on the left side of Figs.~\ref{threemasses} and \ref{trajectory} are examples of "short-(numerical)-answer" problems.
+For the purposes of this project, "multiple choice" and "short-answer" will be considered as separate classes, where short-answer includes numerical answers such as ``$17 kg/m^3$," and formula answers, such as ``\verb!1/2*m*(vx^2+vy^2)!." The problems on the left side of Figs.~\ref{threemasses} and \ref{trajectory} are examples of ``short-(numerical)-answer" problems.
\item[Multiple-choice multiple-response questions] This type of problem, a first step beyond conventional problems, requires a student to evaluate each statement and make a decision about it. The problems Fig.~\ref{problemview} and on the right side of Fig.~\ref{threemasses} are of this type.
\begin{figure}
\includegraphics[width=6.5in]{threemassesjpg}
-\caption{Example of two LON-CAPA problems addressing the same concepts. The problem on the left is a conventional short-numerical-answer problem, while the problem on the right is of type "multiple-choice multiple-response."\label{threemasses}}
+\caption{Example of two LON-CAPA problems addressing the same concepts. The problem on the left is a conventional short-numerical-answer problem, while the problem on the right is of type ``multiple-choice multiple-response."\label{threemasses}}
\end{figure}
\item[Representation-translation questions] This type of problem requires a student to translate between different representations of the same situation, for example from a graphical to a numerical or textual representation. The answer might be given in different formats, for example in the problem on the right side of Fig.~\ref{trajectory}, it is a short-numerical-answer. Translation between representations can be surprisingly challenging for physics learners~\cite{mcdermott,beichner}.
-For the purposes of this project, "representation-translation" will be considered a feature, which may or may not apply to any of the other problem types.
+For the purposes of this project, ``representation-translation" will be considered a feature, which may or may not apply to any of the other problem types.
\begin{figure}
\includegraphics[width=6.5in]{trajectoryjpg}
-\caption{Example of two LON-CAPA problems addressing the same concepts in two different representations. The problem on the left is a conventional short-numerical-answer problem, while the problem on the right requires "representation-translation."\label{trajectory}}
+\caption{Example of two LON-CAPA problems addressing the same concepts in two different representations. The problem on the left is a conventional short-numerical-answer problem, while the problem on the right requires ``representation-translation."\label{trajectory}}
\end{figure}
\item[Ranking-tasks] This type of problem requires a student to rank a number of statements, scenarios, or objects with respect to a certain feature. For example, a student might be asked to rank a number of projectiles in the order that they will hit the ground, or a number of points in order of the strength of their local electric potential.
-\item[Context-based reasoning problems] The distinguishing characteristic of these problems is that they are set in the context of real-world scenarios and not in the context of the artificial "zero-friction" laboratory scenarios of typical textbook problems.
+\item[Context-based reasoning problems] The distinguishing characteristic of these problems is that they are set in the context of real-world scenarios and not in the context of the artificial ``zero-friction" laboratory scenarios of typical textbook problems.
-As in the case of "representation-translation," "context-based-reasoning" in this project will be considered a feature, which may apply or may not apply to any of the other problem types.
-\item[Estimation problems], also known as "Fermi Problems," require the student to form a model for a scenario, and make reasonable assumptions. A typical example is "How many barbers are there in Chicago?" or "How long will I have to wait to find a parking spot?" Students do need to explain their reasoning.
+As in the case of ``representation-translation," ``context-based-reasoning" in this project will be considered a feature, which may apply or may not apply to any of the other problem types.
+\item[Estimation problems], also known as ``Fermi Problems," require the student to form a model for a scenario, and make reasonable assumptions. A typical example is ``How many barbers are there in Chicago?" or ``How long will I have to wait to find a parking spot?" Students do need to explain their reasoning.
While students find it initially hard to believe that these questions have anything to do with physics, hardly any expert physicist would deny their significance in learning how to solve problems~\cite{mazur96}.
A component of this project (see Sect.~\ref{platform}) will be to find and implement mechanisms to implement these question-types within an online system in an authentic yet scalable fashion.
-\item[Qualitative questions] This type of questions asks students to make judgments about physical scenarios, and in that respect are somewhat similar to ranking questions. While the questions themselves are of the type "Is this high enough?" or "Can we safely ignore \ldots?," they often do require at least "back-of-the-envelope" calculations to to give informed answers. As in the case of estimation problems, students do have to explain their reasoning, but the question itself is usually more structured, and at least the initial answer is more easily evaluated by a computer.
-\item[Essay questions] These are "explain why" questions. A certain scenario is presented, and students are asked to explain why it turns out the way it does. Students are not asked to recall a certain law --- it is given to them. Instead, they are asked to discuss its validity.
+\item[Qualitative questions] This type of questions asks students to make judgments about physical scenarios, and in that respect are somewhat similar to ranking questions. While the questions themselves are of the type ``Is this high enough?" or ``Can we safely ignore \ldots?," they often do require at least ``back-of-the-envelope" calculations to to give informed answers. As in the case of estimation problems, students do have to explain their reasoning, but the question itself is usually more structured, and at least the initial answer is more easily evaluated by a computer.
+\item[Essay questions] These are ``explain why" questions. A certain scenario is presented, and students are asked to explain why it turns out the way it does. Students are not asked to recall a certain law --- it is given to them. Instead, they are asked to discuss its validity.
\end{description}
\begin{table}
@@ -272,15 +272,15 @@
Repre\-sentation-translation&AS&AMS&AS&AMS&AS&AS&&&\\\hline
Context-based&MS&AMS&MS&AMS&MS&AMS&GMS&GMS&GMS\\\hline
\end{tabular}
-\caption{Classification scheme for question types, adapted from Redish~\cite{redish}, see Sect.~\ref{class}. The symbols denote different components of the project, i.e., "A" - additional analysis tool development (Sect.~\ref{analysisnew}); "G" - additional scalable grading tool development (Sect.~\ref{platform}); "M" - additional materials development (Sect.~\ref{matdev}); "S" - this question type will be included in the study of its impact (sections~\ref{hypo} and \ref{analysis}).\label{classification}}
+\caption{Classification scheme for question types, adapted from Redish~\cite{redish}, see Sect.~\ref{class}. The symbols denote different components of the project, i.e., ``A" - additional analysis tool development (Sect.~\ref{analysisnew}); ``G" - additional scalable grading tool development (Sect.~\ref{platform}); ``M" - additional materials development (Sect.~\ref{matdev}); ``S" - this question type will be included in the study of its impact (sections~\ref{hypo} and \ref{analysis}).\label{classification}}
\end{table}
\section{Preliminary Project Components}
Several aspects of the research project can be started in the first year, while others will require additional materials development or platform functionality.
-In Table~\ref{classification}, problem types which are marked "S" with no additional symbols have both sufficient functionality support in the LON-CAPA system, and a sufficient library of problems of this type to conduct the study. Problem types marked "M" need additional material to provide a representative sample (Sect.~\ref{matdev}), those marked "G" additional grading tools (Sect.~\ref{platform}), and those marked "A" additional analysis tools (Sect.~\ref{analysisnew}).
+In Table~\ref{classification}, problem types which are marked ``S" with no additional symbols have both sufficient functionality support in the LON-CAPA system, and a sufficient library of problems of this type to conduct the study. Problem types marked ``M" need additional material to provide a representative sample (Sect.~\ref{matdev}), those marked ``G" additional grading tools (Sect.~\ref{platform}), and those marked ``A" additional analysis tools (Sect.~\ref{analysisnew}).
\subsection{Additional Materials Development}\label{matdev}
-For the question types marked "M" in Table~\ref{classification}, the currently existing library of LON-CAPA problems does not provide enough samples to carry out the study. Since development of completely new problems would constitute a project by itself, this component of the current project will heavily draw on existing problem collections, i.e., Redish (\cite{redish}, resource CD), McDermott~\cite{mcdermottprob}, Mazur~\cite{mazur}, and Project Galileo~\cite{galileo}. These research-based problems will be adapted and implemented in the the LON-CAPA system, and new problems only developed where necessary.
+For the question types marked ``M" in Table~\ref{classification}, the currently existing library of LON-CAPA problems does not provide enough samples to carry out the study. Since development of completely new problems would constitute a project by itself, this component of the current project will heavily draw on existing problem collections, i.e., Redish (\cite{redish}, resource CD), McDermott~\cite{mcdermottprob}, Mazur~\cite{mazur}, and Project Galileo~\cite{galileo}. These research-based problems will be adapted and implemented in the the LON-CAPA system, and new problems only developed where necessary.
The goal is to have 12 problems representing each type (Table~\ref{classification}) in each semester, as evenly as possible distributed over the 16 weeks of the semester.
@@ -288,21 +288,21 @@
The proposal budget includes a half-time computer programmer position in the first four years to assist in the implementation of the following additional platform features:
\subsubsection{Scalable Functionality for Manual Grading of Free-Form Answers}\label{platform}
LON-CAPA already offers grading support for free-form student submission, such as
-keyword-highlighting and plagiarism-checks. Additional tools will be developed for the grading of the problem types marked "G" in Table~\ref{classification}: for questions that require student submissions of the type "Explain your reasoning," better coupling between the computer- and manually-evaluated sections will be provided, for the free-form "essay" submissions, better tools to compare student submissions with each other and with exemplary essays.
+keyword-highlighting and plagiarism-checks. Additional tools will be developed for the grading of the problem types marked ``G" in Table~\ref{classification}: for questions that require student submissions of the type "Explain your reasoning," better coupling between the computer- and manually-evaluated sections will be provided, for the free-form ``essay" submissions, better tools to compare student submissions with each other and with exemplary essays.
\subsubsection{Additional Analysis Tools}\label{analysisnew}
-While the premise of this project is that feedback on formative assessment is crucial for the learner, it is almost equally important to the instructor~\cite{pellegrino}, with technology as enabler~\cite{novak,feedback}. Particularly in the context of a research project on formative assessment, timely and comprehensive feedback on student performance --- including new material (Sect.~\ref{matdev}) --- is essential. The LON-CAPA system already has sophisticated analysis tools (see Sect.~\ref{anatool}), but these do not support all questions types in Table~\ref{classification} equally well, and the project includes a tools development component to further enhance these mechanisms for the problem types marked "A."
+While the premise of this project is that feedback on formative assessment is crucial for the learner, it is almost equally important to the instructor~\cite{pellegrino}, with technology as enabler~\cite{novak,feedback}. Particularly in the context of a research project on formative assessment, timely and comprehensive feedback on student performance --- including new material (Sect.~\ref{matdev}) --- is essential. The LON-CAPA system already has sophisticated analysis tools (see Sect.~\ref{anatool}), but these do not support all questions types in Table~\ref{classification} equally well, and the project includes a tools development component to further enhance these mechanisms for the problem types marked ``A."
Data collection on a particular problem type can proceed independently from the existence of the respective analysis tools, since LON-CAPA permanently stores all data.
\section{Research Methodology}\label{analysis}
-\subsection{Establishment of Initial Conditions}Many educational studies result in "no-significant-difference"~\cite{russell}, and particularly the study of question type effectiveness (Sect.~\cite{effect}) may well yield the same result. Many variables may influence the impact of a particular sample of representative problems of a particular type for a particular learner, and it is imperative to understand as much of the "initial conditions" as possible, since the validity of the hypothesis may depend on them.\subsubsection{Learner Attitudes, Beliefs, and Expectations}Instruments have been developed to assess epistemological beliefs, for example the Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument~\cite{EBAPS}. Related to epistemological beliefs are learnerÕs expectations and attitudes, and of particular interest is the Maryland Physics Expectations (MPEX) survey~\cite{MPEX}.\subsubsection{Learner Knowledge about the Topic}\label{prepost}We will use existing concept inventory surveys as both pre- and post-tests.The qualitative Force Concept Inventory~\cite{fci} and the quantitative companion Mechanical Baseline Test~\cite{hestenesmech} have been used in a large number of studies connected to the teaching of introductory mechanics. The Foundation Coalition has been developing a number of relevant concept inventories~\cite{foundation}, namely the Thermodynamics Concept Inventory, the Dynamics Concept Inventory, and the Electromagnetics Concept Inventory (with two subcomponents, namely Waves and Fields). Since these were designed from an engineering point of view, some adjustment might be necessary. In addition, the Conceptual Survey of Electricity and Magnetism (CSEM)~\cite{maloney} is available for the second semester course.
+\subsection{Establishment of Initial Conditions}Many educational studies result in ``no-significant-difference"~\cite{russell}, and particularly the study of question type effectiveness (Sect.~\ref{effect}) may well yield the same result. Many variables may influence the impact of a particular sample of representative problems of a particular type for a particular learner, and it is imperative to understand as much of the ``initial conditions" as possible, since the validity of the hypothesis may depend on them.\subsubsection{Learner Attitudes, Beliefs, and Expectations}Instruments have been developed to assess epistemological beliefs, for example the Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument~\cite{EBAPS}. Related to epistemological beliefs are learnerÕs expectations and attitudes, and of particular interest is the Maryland Physics Expectations (MPEX) survey~\cite{MPEX}.\subsubsection{Learner Knowledge about the Topic}\label{prepost}We will use existing concept inventory surveys as both pre- and post-tests.The qualitative Force Concept Inventory~\cite{fci} and the quantitative companion Mechanical Baseline Test~\cite{hestenesmech} have been used in a large number of studies connected to the teaching of introductory mechanics. The Foundation Coalition has been developing a number of relevant concept inventories~\cite{foundation}, namely the Thermodynamics Concept Inventory, the Dynamics Concept Inventory, and the Electromagnetics Concept Inventory (with two subcomponents, namely Waves and Fields). Since these were designed from an engineering point of view, some adjustment might be necessary. In addition, the Conceptual Survey of Electricity and Magnetism (CSEM)~\cite{maloney} is available for the second semester course.
\subsubsection{Problem Difficulty and Baseline Statistical Data}LON-CAPA automatically keeps tracks of the average number of attempts until a problem is solved, as well as the degree of difficulty and the degree of discrimination. This data is cumulative across semesters, and already exists for all assessment problems from their deployment in previous semesters.
\subsection{Observables}\subsubsection{Effectiveness}\label{effect}Effectiveness will be measured both in terms of performance on summative assessments (quizzes and exams) and on pre-/post-test concept inventory surveys (Sect.~\ref{prepost}). Each item on these instruments will be associated with topically corresponding formative online exercises to determine correlations and differential gain between the feedback types used with the respective online problems. A second posttest, correlated with first semester problems, will be administered at the end of the second semester to determine long-term effects.\subsubsection{Problem Solving Technique}We intend to focus on a subset of students in the LBS Collaborative Learning Laboratory, and observe them while solving problems. Schoenfeld~\cite{schoenfeld} and Foster~\cite{foster} developed instruments to categorize and document the stages and expertlike
-characteristics~\cite{chi} of observed problem-solving activity by learners, as well as application of metacognitive skills.In addition, for all students, log data will be analyzed. Kotas~\cite{kotas} and Minaei~\cite{minaei} developed a mechanism for this log data analysis, which include submission times between attempts, and quality of the entered input. \subsubsection{Help-Seeking Behavior and Discussions}\label{discussion}It is impossible to observe all on-demand help seeking, but interactions in several settings can be analyzed:Online discussions and email communication are preserved within LON-CAPA and can be analyzed even in retrospect for past semesters with respect to relevant behavioral patterns. Table~\ref{discussionex} shows excerpts of discussions around the two problems in Fig.~\ref{trajectory}.
+characteristics~\cite{chi} of observed problem-solving activity by learners, as well as application of metacognitive skills.In addition, for all students, log data will be analyzed. Kotas~\cite{kotas} and Minaei~\cite{minaei} developed a mechanism for this log data analysis, which include submission times between attempts, and quality of the entered input. \subsubsection{Help-Seeking Behavior and Discussions}\label{discussion}It is impossible to observe all help-seeking, but interactions in several settings can be analyzed:Online discussions and email communication are preserved within LON-CAPA and can be analyzed even in retrospect for past semesters with respect to relevant behavioral patterns. Table~\ref{discussionex} shows excerpts of online discussions around the two problems in Fig.~\ref{trajectory}.
\begin{table}
\tiny
@@ -464,20 +464,20 @@
\subsection{Undergraduate}\label{undergrad}
Students in the courses will be informed about the goals and methods of this project beyond the requirements of the consent procedures. Groups of students will be given the opportunity to participate in focus groups, for which stipends are provided in the budget. The undergraduate teaching assistants in this course will be given the option to participate in the data collection and analysis efforts of the project. Research findings will be made available to the students.
\subsection{Graduate}
-The proposal budget includes funds for a graduate student assistantship to to assist in the materials development effort, as well as to collect and analyze data, and work on publication and dissemination. It is expected that the student will base his or her doctoral work on this project. The idea of offering a doctoral degree in Science Education has been considered at MSU, yet at this point in time, the degree would be conferred by the Department of Physics and Astronomy. The doctoral committee for the student will be comprised of members of the Department of Physics and Astronomy and the Division of Science and Mathematics Education.
+The budget includes funds for a graduate student assistantship to assist in the materials development effort, as well as to collect and analyze data, and work on publication and dissemination. It is expected that the student will base his or her doctoral work on this project. The idea of offering a doctoral degree in Science Education has been considered at MSU, yet at this point in time, the degree would be conferred by the Department of Physics and Astronomy. The doctoral committee for the student will be comprised of members of the Department of Physics and Astronomy and the Division of Science and Mathematics Education.
\section{Evaluation}
-The LON-CAPA Faculty Advisory Board was formed as part of our NSF ITR grant project. It consists of eight actively teaching faculty and administrators from a number of colleges on campus of MSU, and meets once every month to both evaluate and advise projects connected to LON-CAPA. We propose to continue using this existing structure to evaluate this projectÕs progress and findings. In addition, Dr.~Kortemeyer's Mentoring Committee, which consists of senior faculty members from both LBS and DSME, will guide and advise the progress of this project.
+The LON-CAPA Faculty Advisory Board was formed as part of our NSF ITR grant project. It consists of eight actively teaching faculty members and administrators from a number of colleges on campus of MSU, and meets once every month to both evaluate and advise projects connected to LON-CAPA. We propose to continue using this existing structure to evaluate this projectÕs progress and findings. In addition, Dr.~Kortemeyer's Mentoring Committee, which consists of senior faculty members from both LBS and DSME, will guide and advise the progress of this project.
\section{Dissemination}\label{dissem}
-We will present papers at conferences such as the LON-CAPA User Conference, IEEE Frontiers in Education, Educause/NLII, Sloan C, the annual meetings of the Deutsche Physikalische Gesellschaft and the Gesellschaft f\"ur Didaktik der Chemie und Physik, the European Workshop for Multimedia in Physics Education, the Conference on Computer Based Learning in Science, and the American Association of Physics Teachers Annual Meeting. Dr. Kortemeyer presented at these conferences before. We will submit papers to journals such as The Physics Teacher, the American Journal of Physics, Computers and Education, and the Journal of Asynchronous Learning Networks. Finally, any content material adapted and implemented in this project will be immediately available to all participant LON-CAPA institutions, and via the LON-CAPA gateway to the NSF-funded National Science Digital Library. Any mature additional platform functionality will be made available in the production releases of the open-source freeware LON-CAPA system.
+We will present papers at conferences such as the LON-CAPA User Conference, IEEE Frontiers in Education, Educause/NLII, Sloan C, the European Workshop for Multimedia in Physics Education, the Conference on Computer Based Learning in Science (Dr. Kortemeyer presented at these conferences before), the annual meetings of the Deutsche Physikalische Gesellschaft and the Gesellschaft f\"ur Didaktik der Chemie und Physik, and the American Association of Physics Teachers Annual and PERC Meetings. We will submit papers to journals such as The Physics Teacher, the American Journal of Physics, Computers and Education, and the Journal of Asynchronous Learning Networks. Finally, any content material adapted and implemented in this project will be immediately available to all participating LON-CAPA institutions, and via the LON-CAPA gateway to the NSF-funded National Science Digital Library. Any mature additional platform functionality will be made available in the production releases of the open-source freeware LON-CAPA system.
\section{Professional Development and Mentoring}
Besides participating in conferences and workshops (Sect.~\ref{dissem}), the proposal includes funds to visit Physics Education Research groups in the US and Germany. Dr.~Kortemeyer already visited the groups at Maryland (Redish and Hammer) and Oldenburg (Hilf), and is hoping to be able to conduct similar visits to for example Harvard (Mazur), Washington (McDermott), Arizona (Hestenes), Minnesota (Heller), Bremen (Schecker), and Kaiserlautern (Jodl).
-During the first years of this project, Dr. Kortemeyer will be teaching the lecture and laboratory courses (Sect.~\ref{coursesdesc}) together with Dr. Walter Benenson, a University Distinguished Professor of Physics, who will interact with Dr. Kortemeyer on a daily base.
+In addition to guidance from the Mentoring Committee, during the first years of this project, Dr. Kor\-temeyer will be teaching the lecture and laboratory courses (Sect.~\ref{coursesdesc}) together with Dr. Walter Benenson, a University Distinguished Professor of Physics, who will interact with Dr. Kortemeyer on a daily base.
\section{Project Timeline}
The timeline for the project is outlined in Table~\ref{timeline}.
-.
+
\begin{table}
\begin{tabular}{|l|p{3.2cm}|p{3.2cm}|p{3.2cm}|p{3.2cm}|}
\hline
--www1090178715--