[LON-CAPA-cvs] cvs: modules /gerd/concept description.tex

[www]

This is a MIME encoded message

--www1089472253
Content-Type: text/plain

www		Sat Jul 10 11:10:53 2004 EDT

  Modified files:              
    /modules/gerd/concept	description.tex 
  Log:
  Next round.
  
  
--www1089472253
Content-Type: text/plain
Content-Disposition: attachment; filename="www-20040710111053.txt"

Index: modules/gerd/concept/description.tex
diff -u modules/gerd/concept/description.tex:1.9 modules/gerd/concept/description.tex:1.10
--- modules/gerd/concept/description.tex:1.9	Fri Jul  9 15:20:56 2004
+++ modules/gerd/concept/description.tex	Sat Jul 10 11:10:53 2004
@@ -77,7 +77,7 @@
 
 One cannot really blame learners for short-circuiting physics "learning" this way, since the cognitive and metacognitive skills, which physicists value so highly, are hardly ever made explicit, neither in instruction, nor in formative or summative assessment~\cite{lin,reif,mazur96}; in fact, they are mostly altogether "hidden"~\cite{redish} from all aspects of a course, and students are affirmed in their novice expectations~\cite{hammer} of what it is to "do physics." To quote Lin~\cite{lin}: "The primary determinants of student performance are the specific tasks for which teachers explicitly hold student responsible (e.g. problem sets and exams), rather than the general goals of the teacher (e.g. conveying an appreciation of the power of physics in a broad context)."
 
-\subsection{The Problem with Problems}
+\subsection{The Problem with Problems}\label{hypo}
 {\bf This project focusses on formative assessment in introductory physics education, and how formative assessment can be used to help learners re-evaluate their epistemologies, develop expertlike problem solving skills, and gain a conceptual understanding of physics.} The challenge is to move students away from treating physics as a set of unrelated factoids and formulas, as well as away from focussing on memorizing and using formulas without interpretation or sense-making~\cite{hammer}, and toward "thinking like a physicist."
 
 
@@ -88,11 +88,11 @@
 
 Alternative formative assessment as a classroom tool, where students are forced to verbally  express their views and teach each other, rather than calculate answers~\cite{mazur}, is starting to be adopted as an effective teaching practice in more and more courses. Formative assessment outside the classroom, on the other hand, is frequently limited by logistics, particularly in large enrollment courses, where timely feedback is often impossible without the use of computerized systems (e.g.~\cite{thoennessen,kashy00}). {\bf This project will focus on online homework as formative assessment tool.} As a model system, the project will use the Learning{\it Online} Network with CAPA (LON-CAPA), described in section~\ref{loncapa}.
 
-The assumption of this project is that "the problem with problems" (a phrase borrowed from~\cite{mazur96}) is that
+The project assumes that "the problem with problems" (a phrase borrowed from~\cite{mazur96}) is that
 \begin{quote}
 {\bf Hypothesis 1a:} Standard calculation-oriented textbook problems affirm non-expertlike epistemologies and encourage non-expertlike problem-solving strategies
 \end{quote}
-and that by the reverse token --- within the limitations of online computer-evaluated problems ---
+A further assumption is that by the reverse token
 \begin{quote}
 {\bf Hypothesis 1b:} There are types of online formative assessment computer-evaluated problems which make learners confront their non-expertlike epistemologies and encourage expertlike problem-solving strategies\end{quote}
 
@@ -128,9 +128,9 @@
 Starting August 2004, Dr.~Kortemeyer will be working in a tenure-track position as Assistant Professor of Physics Education. His appointment will be split 75/25\% between the Lyman Briggs School of Science (LBS) and DSME. He will also be holding an appointment as Adjunct Professor of Physics in the Department of Physics and Astronomy. His teaching responsibilities will include the introductory calculus-based physics sequence (lecture and lab) in LBS, as well as seminars in special topics. His research will be focused on postsecondary science teaching and learning, with a special emphasis on the use of technology.\subsection{Michigan State University}Michigan State University is one of the earliest land-grant institutions in the United States. MSU is committed to providing equal educational opportunity to all qualified applicants; at the undergraduate level, the university offers comprehensive programs in the liberal arts and sciences, and provides opportunities for students of varying interests, abilities, backgrounds, and expectations.  The total enrollment is approximately 44,000, 35,000 of which are undergraduates. 54\% of the student population are women, 8.1\% African American, 5.1\% Asian/Pacific Islander, 2.8\% Chicano/Other Hispanic, and 0.6\% Native American. Of the freshman class, the average high school GPA is 3.58.\subsection{Lyman Briggs School of Science}The Lyman Briggs School (LBS) at Michigan State University is a residential learning community devoted to studying the natural sciences and their impact on society. All under one roof, LBS encompasses physics, chemistry, biology, and computer laboratories; classrooms; faculty, administrative, and academic support staff offices; student residences; a dining hall; and a convenience store. With approximately 1500 students, LBS offers the benefits of a small, liberal arts college with the resources of a large research university.\subsection{Division of Science and Mathematics Education}The Division of Science and Mathematics Education (DSME) was founded at Michigan State University in 1989, and is co-administered by the College of Natural Science and the College of Education. Academic specialists and faculty members with partial appointments in various departments and other colleges, graduate and undergraduate students, and professional and clerical staff work together in DSME to conduct a variety of research projects, as well as to offer courses, degree programs, and other activities in support of its mission.
 \subsection{Model System: The Learning{\it Online} Network with CAPA}\label{loncapa}For several aspects of the proposed project, the Learning{\it Online} Network with Computer-Assisted Personalized Approach (LON-CAPA; {\tt http://www.lon-capa.org/}) will be the model system. LON-CAPA is a distributed learning content management, course management, and assessment system, and also the model system of the current NSF-ITR grant, see section~\cite{results}. 
 
-LON-CAPAÕs core development group is located at Michigan State University, and in addition to faculty members, has a staff of three fulltime programmers, two user support staff, one technician, one graduate student, and one project coordinator. LON-CAPA is open-source (GNU General Public License) freeware, there are no licensing costs associated. The LON-CAPA group offers training and support, as well as hosting.
+LON-CAPAÕs core development group is located at Michigan State University, and in addition to faculty members, has a staff of three fulltime programmers, two user support staff, one technician, one graduate student, and one project coordinator. The LON-CAPA group also offers training and support for adopters of the system.
 
-Dr.~Kortemeyer is the director of the LON-CAPA project. \subsubsection{Shared Distributed Content Repository}LON-CAPA is designed around the concept of easy sharing and re-use of learning resources. The ITR research showed that this is only possible by integrating all layers of the infrastructure to give instructors a seamless user experience: pure library systems, such as NSDL, do not offer instructors the ability to in one system: locate existing content; generate and publish new content; enforce secure digital rights management; assemble (sequence) content; deploy the assembled content in a complete course management system; and have continual objective and subjective assessment of the learning content quality.In addition, the system has to be highly scalable, and avoid single points of failure.In LON-CAPA, the underlying distributed content repository spans all servers in a given cluster. Navigation through selected resources is provided by an internal sequencing tool, which allows assembling, re-using, and re-purposing content at different levels of granularity (pages, lessons, modules, chapters, etc). LON-CAPA provides highly customizable access control on resources, and has a built-in key mechanism to charge for content access. The shared content pool of LON-CAPA currently contains over 60,000 learning resources, including more than 18,000 personalized homework problems. Any content material contributed ot the pool is immediately available ready-to-use within the system at all participating sites, thus facilitating dissemination of curricular development efforts. Disciplines include astronomy, biology, business, chemistry, civil engineering, computer science, family and child ecology, geology, human food and nutrition, human medicine, mathematics, medical technology, physics, and psychology. A large fraction of these resources are also available through the gateway to the National Science Digital Library. In addition, the problem supplements to a number of commercial textbooks are available in LON-CAPA format.The network provides constant assessment of the resource quality through objective and subjective dynamic metadata. Selection of a learning resource by instructors at other institutions while constructing a learning module does both establish a de-facto peer-review mechanism and provide additional context information for each resource. In addition, access statistics are being kept, and learners can put evaluation information on each resources.\subsubsection{Formative and Summative Assessment Capabilities}LON-CAPA started in 1992 as a system to give personalized homework to students in introductory physics courses.  ÒPersonalized" means that each student sees a different version of the same computer-generated problem: different numbers, choices, graphs, images, simulation parameters, etc, Fig.~\ref{twoproblems}.
+LON-CAPA is open-source (GNU General Public License) freeware, there are no licensing costs associated. Both aspects are important for the success of a research project like the one proposed here. The open-source nature of the system allows researchers to modify and adapt the system in order to address the needs of their project, and the freeware character allows easier dissemination of results, in particular adaptation and implementation at other universities.\subsubsection{Shared Distributed Content Repository}LON-CAPA is designed around the concept of easy sharing and re-use of learning resources. The ITR research showed that this is only possible by integrating all layers of the infrastructure to give instructors a seamless user experience: pure library systems, such as NSDL, do not offer instructors the ability to in one system: locate existing content; generate and publish new content; enforce secure digital rights management; assemble (sequence) content; deploy the assembled content in a complete course management system; and have continual objective and subjective assessment of the learning content quality.In addition, the system has to be highly scalable, and avoid single points of failure.In LON-CAPA, the underlying distributed content repository spans all servers in a given cluster. Navigation through selected resources is provided by an internal sequencing tool, which allows assembling, re-using, and re-purposing content at different levels of granularity (pages, lessons, modules, chapters, etc). LON-CAPA provides highly customizable access control on resources, and has a built-in key mechanism to charge for content access. The shared content pool of LON-CAPA currently contains over 60,000 learning resources, including more than 18,000 personalized homework problems. Any content material contributed ot the pool is immediately available ready-to-use within the system at all participating sites, thus facilitating dissemination of curricular development efforts. Disciplines include astronomy, biology, business, chemistry, civil engineering, computer science, family and child ecology, geology, human food and nutrition, human medicine, mathematics, medical technology, physics, and psychology. A large fraction of these resources are also available through the gateway to the National Science Digital Library. In addition, the problem supplements to a number of commercial textbooks are available in LON-CAPA format.The network provides constant assessment of the resource quality through objective and subjective dynamic metadata. Selection of a learning resource by instructors at other institutions while constructing a learning module does both establish a de-facto peer-review mechanism and provide additional context information for each resource. In addition, access statistics are being kept, and learners can put evaluation information on each resources.\subsubsection{Formative and Summative Assessment Capabilities}LON-CAPA started in 1992 as a system to give personalized homework to students in introductory physics courses.  ÒPersonalized" means that each student sees a different version of the same computer-generated problem: different numbers, choices, graphs, images, simulation parameters, etc, Fig.~\ref{twoproblems}.
 \begin{figure}
 \includegraphics[width=6.5in]{atwood}
 \caption{Web-rendering of the same LON-CAPA problem for two different students.\label{twoproblems}
@@ -151,16 +151,40 @@
 In addition to having whiteboards and wireless laptop computers for students to work with in flexible group settings, the facility will have integrated observation equipment to video- and audio-record student interactions. All recorded information is immediately digitized and made available for transcription and analysis using the Transana~\cite{transana} software system.
 \subsection{Courses}
 The project will be carried out  LBS course sequence LBS 271/272 (calculus-based physics I \& II), which have an enrollment of over 200 students each semester. 
-\section{Classification of Online Formative Assessment Problems}
+\subsection{Synergy between Project and Institutional Goals}
+The mission statement of the Lyman-Briggs School of Science includes the statement
+\begin{quote}
+Curious about the natural world around them, LBS's highly motivated students and faculty work together to learn what is known and explore what is new.
+\end{quote}
+The mission statement of the Division of Science and Mathematics Education is
+\begin{quote}
+to improve science and mathematics education, from kindergarten through the undergraduate years, through the professional development of preservice and inservice teachers and faculty members. 
+\end{quote}
+These ambitious goals clearly mean going beyond a surface-recapitulation of disconnected factoids, toward a deep appreciation of natural phenomena and their inter-connectedness. Research and teaching are closely intertwined in this project, with the goal of improving the learning and teaching
+of the natural sciences, even as the study itself is being carried out. The outcomes of this study will inform faculty members, and aligns with the goals of the Division of Science and Mathematics Education.
+\section{Classification of Online Formative Assessment Problems}\label{class}
 Redish~\cite{redish} distinguishes eight types of exam and homework questions, an adapted version of which will form the general classification scheme for Hypothesis 1b:
 \begin{description}
 \item[Multiple-choice and short-answer questions] The most basic and most easily computer-evaluated type of question, representing the typical back-of-chapter textbook problem.
 
 For the purposes of this project, "multiple choice" and "short-answer" will be considered as separate classes, where short-answer includes numerical answers such as "$17 kg/m^3$," and formula answers, such as "\verb!1/2*m*(vx^2+vy^2)!."  The problems on the left side of Figs.~\ref{threemasses} and \ref{trajectory} are examples of "short-(numerical)-answer" problems.
 \item[Multiple-choice multiple-response questions]  This type of problem requires a student to evaluate each statement and make a decision about it. The problem on the right side of Fig.~\ref{threemasses} is of this type.
+
+
+\begin{figure}
+\includegraphics[width=6.5in]{threemassesjpg}
+\caption{Example of two LON-CAPA problems addressing the same concepts. The problem on the left is a traditional short-numerical-answer problem, while the problem is of type "multiple-choice multiple-response."\label{threemasses}}
+\end{figure}
+
 \item[Representation-translation questions] This type of problem requires a student to translate between different representations of the same situation, for example from a graphical to a numerical or textual representation. The answer might be given in different formats, for example in the problem on the right side of Fig.~\ref{trajectory}, it is a short-numerical-answer.
 
 For the purposes of this project, "representation-translation" will be considered a feature, which may apply or may not apply to any of the other problem types.
+
+\begin{figure}
+\includegraphics[width=6.5in]{trajectoryjpg}
+\caption{Example of two LON-CAPA problems addressing the same concepts in two different representations. The problem on the left is a traditional short-numerical-answer problem, while the problem is of type "multiple-choice multiple-response."\label{trajectory}}
+\end{figure}
+
 \item[Ranking-tasks] This type of problem requires a student to rank a number of statements, scenarios, or objects with respect to a certain feature. For example, a student might be asked to rank a number of projectiles in the order that they will hit the ground, or a number of points in order of the strength of their local electric potential.
 \item[Context-based reasoning problems] The distinguishing characteristic of these problems is that they are set in the context of real-world scenarios and not in the context of the artificial "zero-friction" laboratory scenarios of typical textbook problems.
 
@@ -169,10 +193,11 @@
 
 While students find it initially hard to believe that these questions have anything to do with physics, hardly any expert physicist would deny their significance in learning how to solve problems~\cite{mazur96}. 
 
-A component of this project (see section~\ref{estimate}) will be to find and implement mechanisms to implement these question-types within an online system in an authentic yet scalable fashion.
+A component of this project (see section~\ref{platform}) will be to find and implement mechanisms to implement these question-types within an online system in an authentic yet scalable fashion.
 \item[Qualitative questions] This type of questions asks students to make judgments about physical scenarios, and in that respect are somewhat similar to ranking questions. While the questions themselves are of the type "Is this high enough?" or "Can we safely ignore \ldots?," they often do require at least "back-of-the-envelope" calculations to to give informed answers. As in the case of estimation problems, students do have to explain their reasoning, but the question itself is usually more structured, and at least the initial answer is more easily evaluated by a computer.
 \item[Essay questions] These are "explain why" questions. A certain scenario is presented, and students are asked to explain why it turns out the way it does. Students are not asked to recall a certain law --- it is given to them. Instead, they are asked to discuss its validity.
 \end{description}
+
 Table~\ref{classification} summarizes the classification scheme which will be used in this project.
 \begin{table}
 \small
@@ -181,24 +206,23 @@
 &\multicolumn{4}{|c|}{Multiple-choice and short-answer}&Multiple-choice multiple-response&Ranking&Esti\-mation&Quali\-tative&Essay\\
 &Multiple-choice&Tex\-tual&Nume\-rical&For\-mula&&&&&\\
 \hline
-"Traditional"&&&&&&&&&\\\hline
-Repre\-sentation-translation&&&&&&&&&\\\hline
-Context-based&&&&&&&&&\\\hline
+"Traditional"&S&MS&S&MS&S&S&&&\\\hline
+Repre\-sentation-translation&S&MS&S&MS&S&S&&&\\\hline
+Context-based&MS&MS&MS&MS&MS&MS&PMS&PMS&PMS\\\hline
 \end{tabular}
-\caption{Classification scheme for question types, adapted from Redish~\cite{redish}.\label{classification}}
+\caption{Classification scheme for question types, adapted from Redish~\cite{redish}, see section~\ref{class}. The symbols denote different components of the project, i.e., "P" - additional platform development efforts (section~\ref{platform}); "M" - additional materials development (section~\ref{matdev}); "S" - this question type will be included in the study of its impact (sections~\ref{hypo} and \ref{analysis}).\label{classification}}
 \end{table}
+\section{Preliminary Project Components}
+Several aspects of the research project can be started in the first year, while others will require additional functionality in the LON-CAPA platform, and the preparation of additional homework problems of certain types.
 
-\begin{figure}
-\includegraphics[width=6.5in]{threemassesjpg}
-\caption{Example of two LON-CAPA problems addressing the same concepts. The problem on the left is a traditional short-numerical-answer problem, while the problem is of type "multiple-choice multiple-response."\label{threemasses}}
-\end{figure}
-
-\begin{figure}
-\includegraphics[width=6.5in]{trajectoryjpg}
-\caption{Example of two LON-CAPA problems addressing the same concepts in two different representations. The problem on the left is a traditional short-numerical-answer problem, while the problem is of type "multiple-choice multiple-response."\label{trajectory}}
-\end{figure}
-
-\section{Research Methodology}
+In Table~\ref{classification}, problem types which are marked "S" with no additional symbols have both sufficient functionality support in the LON-CAPA system, and a sufficient library of problems of this type to conduct the study. If a problem type is marked "M," the library of problems of this type is still small, and additional problems need to be developed. If a problem type is marked "P", it means that while the platform in its current version can support it, the solution does not scale well with the number of students due to the effort required for manual grading.
+\subsection{Additional Platform Development}
+\subsubsection{Scalable Functionality for Estimation and Qualitative Questions}\label{platform}
+\subsubsection{Additional Analysis Tools}
+While the premise of this project is that feedback on formative assessment is crucial for the learner, it is almost equally important to the instructor~\cite{pellegrino}, with technology as enabler~\cite{novak,feedback}.
+\subsection{Additional Materials Development}\label{matdev}
+For the question types marked "M" in Table~\ref{classification}, the currently existing library of LON-CAPA problems is not provide enough samples to carry out the study. Within this project, new homework problems of this type will be developed. Since development of completely new problems would constitute a project by itself, this component of the current project will heavily draw on existing problem collections, i.e., Redish (\cite{redish}, resource CD), McDermott~\cite{mcdermottprob}, Mazur~\cite{mazur}, and Project Galileo~\cite{galileo}. These research-based problems will be adapted and implemented in the the LON-CAPA system.
+\section{Research Methodology}\label{analysis}
 \subsection{Establishment of Initial Conditions}The validity of the hypotheses may depend on both learner and assessment characteristics.\subsubsection{Learner Attitudes, Beliefs, and Expectations}Instruments have been developed to assess epistemological beliefs, for example the Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument~\cite{EBAPS}. Related to epistemological beliefs are learnerÕs expectations and attitudes, and of particular interest is the Maryland Physics Expectations (MPEX) survey~\cite{MPEX}.\subsubsection{Learner Knowledge about the Topic}We will use existing concept inventory surveys as both pre- and post-tests.The qualitative Force Concept Inventory~\cite{fci} and the quantitative companion Mechanical Baseline Test~\cite{hestenesmech} have been used in a large number of studies connected to the teaching of introductory mechanics. The Foundation Coalition has been developing a number of relevant concept inventories~\cite{foundation}, namely the Thermodynamics Concept Inventory, the Dynamics Concept Inventory, and the Electromagnetics Concept Inventory (with two subcomponents, namely Waves and Fields).  Since these were designed from an engineering point of view, some adjustment might be necessary. In addition, the Conceptual Survey of Electricity and Magnetism (CSEM)~\cite{maloney} is available for the second semester course.
 
 4.2.3 Assessment LevelOnline problems will be categorized according to BloomÕs taxonomy [Bloom56]. We will distinguished between ÒLevel 1Ó questions, which only require minimal interpretation of the presented content (corresponding to the Knowledge Level of BloomÕs taxonomy) and ÒLevel 2Ó questions (Comprehension and Application), which require learners to make connections between presented material and apply them to more complex scenarios.4.2.4 Assessment Difficulty and Baseline Statistical DataLON-CAPA automatically keeps tracks of the average number of attempts until a problem is solved, as well as the degree of difficulty and the degree of discrimination. This data is cumulative across semesters, and already exists for all assessment problems from their deployment in previous semesters.4.3 Observables4.3.1 EffectivenessEffectiveness will be measured both in terms of performance on summative assessments (quizzes and exams) and on pre-/post-test concept inventory surveys (section 4.2.2).  Each item on these instruments will be associated with topically corresponding formative online exercises to determine correlations and differential gain between the feedback types used with the respective online problems. A second posttest, correlated with first semester problems, will be administered at the end of the second semester to determine long-term effects.4.3.2 Problem Solving TechniqueWe intend to focus on a subset of students in the LBS Collaborative Learning Laboratory, and observe them while solving problems. Schoenfeld [Schoenfeld85] and Foster [Foster00] developed instruments to categorize and document the stages and expertlike characteristics [Chi81] of observed problem-solving activity by learners, as well as application of metacognitive skills.In addition, we will interview a group of students from all courses regarding their problem-solving strategies. Pascarella [Pascarella02] developed some frameworks for these interviews, which can be built upon.Finally, for all students in all courses, LON-CAPA log data will be analyzed. Kotas [Kotas00] and Minaei [Minaei03] developed a mechanism for this log data analysis, which include submission times between attempts, and quality of the entered input. 4.3.3 Help-Seeking BehaviorIt is impossible to observe all on-demand help seeking, but interactions in several settings can be analyzed:¥ Online discussions and email communication are preserved within LON-CAPA and can be analyzed even in retrospect for past semesters with respect to relevant behavioral patterns. Discussion contributions and states can be linked to online transactions, such as submission of homework attempts, browsing of content material, and hint usage. Wallace [Wallace03] reviewed existing research on such online interactions, however, some adaptation of several of the existing conceptualizations will be necessary to account for the nature of science and in particular physics and biology courses.¥ For the subset of LBS students who come to the LBS Collaborative Learning Laboratory, group discussions can to be documented, and linked to online behavior as absolute timing and learner identify are preserved.¥ Minaei [Minaei03] developed data mining strategies to categorize learner behavior, including navigational patterns between assessment and content material, the use of feedback, and communication functions.¥ Self-reporting can be used for several other help-seeking mechanisms, such as textbook use and peer-interaction ([Riffell03a, Riffell03b]).4.3.4 Cross-Cutting Open-Ended Documentation of Learner PerceptionsWe will interview focus groups of students from every course regarding their experiences and perceived relative helpfulness of the different feedback mechanisms. We will ask them to also reflect on how they perceived these mechanisms to influence their problem solving strategies. In the framework of the current NSF ITR grant [NSFITR00] an instrument was developed to gather learner input on perceived time-on-task and perceived helpfulness and functionality of different aspects of the system.4.4 Data Collection and Comparison GroupsFor the data collection, results would be most meaningful, if we divided the course into groups that experience the same assignment with different feedback mechanisms; indeed, this is what we propose to do in later phases of the study regarding the variation of solely the feedback character. However, when varying the number of available tries and the immediacy of the feedback, we will do so concurrently for all students across different problems Ð otherwise, it would be inevitable that students will complain about unfairness compared to the students with more attempts or immediate feedback. Also students with delayed feedback would most certainly confer with students in the other group.
@@ -259,6 +283,14 @@
 \bibitem{ncsu} North Carolina State University, Physics Education and Development Group, {\tt http://www.physics.ncsu.edu:8380/physics\_ed/Room\_Design\_files/frame.htm}
 \bibitem{transana} University of Wisconsin, Wisconsin Center for Education Research, {\tt http://www2.wcer.wisc.edu/Transana/}
 
+% Prelim
+
+\bibitem{novak} Gregor Novak, Andrew Gavrin, Wolfgang Christian, and Evelyn Patterson,
+{\it Just-In-Time Teaching : Blending Active Learning with Web Technology}, Prentice Hall, ISBN 0-13-085034-9 (1999)
+\bibitem{feedback} Matthew Hall, Joyce Parker, Behrouz Minaei-Bidgoli, Guy Albertelli, Gerd Kortemeyer, and Edwin Kashy, {\it Gathering and Timely Use of Feedback from Individualized On-Line Work}, Frontiers in Education (2004)
+\bibitem{mcdermottprob} Lillian C. McDermott, Peter S. Shaffer, and the Physics Education Group at the University of Washington, {\it Tutorials in Introductory Physics, Homework}, Prentice Hall, ISBN 0-13-954629-4 (1998)
+\bibitem{galileo} Project Galileo, {\tt http://galileo.harvard.edu/}
+
 % Methodology
 \bibitem{EBAPS} A. Elby, J. Fredriksen, C. Schwarz, and B. White, {\it Epistemological Beliefs Assessment for Physical Science (EBAPS) Instrument}, {\tt http://www2.physics.umd.edu/$\sim$elby/EBAPS/EBAPS\_items.htm} (2001)
 \bibitem{MPEX} E. F. Redish, J. M. Saul, and R. N. Steinberg, {\it Student expectations in introductory physics}, Am. J. Physics {\bf 66}, 212-224 (1998), survey available on The Physics Suite CD in Teaching Physics, ISBN 0-471-39378-9 (2003)

--www1089472253--