[LON-CAPA-dev] NSF ITR Site Visit

[Helen Keefe]

Here are my notes from the site visit.  I thought all of you should be
aware of what went on.

Dissemination must be done of both product and project.
* FIE Booth, Sloan ALN Conference
* Physics faculty have given invited talks
* Potential collaboration with MultiMedia Physik
* Seminars in Academic Computing
There needs to be more specific articulation of talks and presentations
in reports to NSF.

Reports
There will be three reports, one from NSF, and one from each evaluator
Reports will be delivered in approximately one month
MSU may respond to reports.

Strengths
There was a good collection of people assembled for the Site Visit.
Each group provided a different look at the project.

The technology is well conceived; it has good potential.
The design is scalable and distributable.

There are a variety of problems with immediate feedback.
The use of role-based identities is interesting and intriguing.

There are good ideas on the business model.

There is a lot of development expertise on the system due to the
predecessor systems (CAPA).
There is a critical mass of following.  This is the next step.
There has been good progress with limited resources.
Big decisions need to be made about resources.
Everyone seems to be very dedicated to the project.

Concerns
IP issues across institutions.

Evaluation and Assessment
* It is unwise to wait to create a plan.
* There needs to be more formative, ongoing evaluation
* This is a large project with many tasks – more than one person needs
to be involved.
* Evaluation across sites is the biggest potential.

Usability & User Interface
* The interface needs to be more intuitive for the general user.
* The interface is difficult for entering problems and questions.
* Is there buy-in from faculty?
* There is a trade-off between features and usability.

Overall Cohesiveness Among Groups
* No sense of working together
* No strong leadership
* Suggest that students are observed in doing their homework
* There should be more involvement in the bigger picture.

Relationship to other similar systems
* What are other models within other disciplines?
WebWork
Mathematica
Others?

Homework needs to be more shareable.
* Levels of difficulty for students
* More differentiation
* Use of assessment tools – there should be training for teachers.

Reuse of content for resource sharing
* Sharing rather than reusing
* Pilot projects – show what we’ve done

Recommendations
Put level of difficulty into metadata
Involve users in design and testing
Hire user interface expert
Concentrate on teamwork
Involve College of Education through Associate Dean (Joan)
Add student(s) to Advisory Board
Talk to users
Ongoing testing on
* Usability
* Resource sharing
* Use of product
* Impact on learning

RET
* More involvement with teachers
* Give them specific responsibilities
* Need to have a research component

Find market niche – Physics, Chemistry

Move evaluation to forefront

Freeze LON-CAPA development, and do user studies
Future development should be based on user interaction
There needs to be a formative evaluation.
There needs to be a research component.

Realistic Scope
* Limited niche
* Demonstrate feasibility
* Formative evaluation with ongoing documentation
* Development of user community
* Work as a team


Further discussion and actions:
Write a note to Harriet regarding re-allocation of budget for user
interface programming.

Ask for leverage from NSF to get internal funding.

The system is of great value – intrinsic to the institution.

There is good administrative awareness.

Do it		Demonstrate		Disseminate

Coordination of Research Activity
* Create evaluation user groups
* Leadership
* More involvement with Evaluation Board
* Create tool kits – surveys
* Aggregating reports





-- 
Helen Keefe
LON-CAPA/ ITR Project Manager
Laboratory for Instructional Technology in Education
111 North Kedzie Labs
East Lansing, MI 48824

E-mail: helen@lon-capa.org
Web:   http://www.lite.msu.edu/
Phone: 517-432-9867
Fax:   517-432-5653