[LON-CAPA-admin] Batch creation
Stuart Raeburn
raeburn at msu.edu
Mon Jul 2 13:06:56 EDT 2012
Hi Paul,
Your post asks about four different things: (a) course creation, (b)
Shibboleth SSO, (c) Auto-enrollment, (d) Auto-updates of user
information.
As a consequence my reply is quite long.
(a) course creation
> ... it is probably best to create the courses interactively within
> LON-CAPA rather than trying to generate XML files somewhere else
> for batch creation of the courses. Is that right?
My recommendation is either (a) for the Domain Coordinator to create
courses interactively, one course at a time, or (b) enable course
requests for faculty, and ask instructors to submit course requests,
and then for the Domain Coordinator to either approve them (or
configure them to be created automatically).
There is a trade off between empowering faculty and reducing wait
times for course availability, versus how much data entry you wish to
ask faculty to do, and how much control the Domain Coordinator would
like to have over each course created in the domain.
Generation of the XML for batch creation of courses was the mechanism
used pre-2.9 by domains which wanted to support faculty requests for
LON-CAPA courses via a web form. (MSU used this from 2004 to 2010, at
which point there was a switch to the course request web form included
within LON-CAPA 2.9, and later).
Of course, if an institution decided to use LON-CAPA as its only
Course Management system, then batch creation (using appropriate XML
for each course) to prepopulate LON-CAPA courses for all courses each
semester for each instructor of record would be the likely approach. I
do not know of an institution where currently that is the case.
(b) Shibboleth SSO
> For the Shibboleth part, do we modify localauth.pm?
No, not if you plan to use Single Sign On.
Support in LON-CAPA for Shibboleth Single Sign On has been implemented
for the next LON-CAPA release - 2.11 - (which will be available soon).
I expect to update the Domain Coordination Manual (for 2.11) with
information about Shibboleth. The general approach is configure the
LON-CAPA server in your domain used to handle log-ins as a Shibboleth
2 Service Provider (SP).
To get your server ready for use as a Shibboleth SP for use with
LON-CAPA 2.11 you will need to do the following:
1. Add shibboleth.repo to /etc/yum.repos.d
[security_shibboleth]
name=Shibboleth (CentOS-5)
type=rpm-md
baseurl=http://download.opensuse.org/repositories/security:/shibboleth/CentOS-5/
gpgcheck=1
gpgkey=http://download.opensuse.org/repositories/security:/shibboleth/CentOS_5/repodata/repomd.xml.key
enabled=1
2. Install RPMs
yum install shibboleth mod_ssl
(You may also wish to install perl-LDAP).
3. Either create your own SSL certificate or obtain an SSL certificate
which can be used when your Shibboleth SP authenticates to your
institution's Shibboleth IdP (Identity Provider). Your institution's
IdP will likely require you to send them the contents of the
certificate.
4. Configure /etc/shibboleth/shibboleth2.xml with settings appropriate
for your institution's IdP.
5. Set shibd to start on boot
/sbin/chkconfig shibd on
6. To /etc/httpd/conf/httpd.conf add following:
# Load the Shibboleth module.
LoadModule mod_shib /usr/lib/shibboleth/mod_shib_22.so
immediately before:
Include conf/loncapa_apache.conf
7. Add loncapa_apache_localbsu.conf from to /etc/httpd/conf
That file will contain:
PerlSetVar lonSSOUserLogoutMessageFile
/home/httpd/html/adm/sso_logout_link_html_frag
PerlSetVar lonSSOUserUnknownRedirect /adm/sso_failed_login.html
PerlSetVar lonSSOUserDomain bsu
Example contents for these two files might be:
/home/httpd/html/adm/sso_logout_link_html_frag
<p>
As your original log-in to LON-CAPA was authenticated by BSU's central
Shibboleth Single Sign On service, your Shibboleth credentials are
still valid.<br />
Until you close your web browser, BSU web applications which support
Shibboleth Single Sign-on (including LON-CAPA) will not require you to
re-enter your BSU credentials.</p>
<p>
To expire your active Shibboleth authentication token you must quit
your web browser.
</p>
/home/httpd/html/adm/sso_failed_login.html
<html>
<head>
<title>LON-CAPA SSO, Unknown User</title>
</head>
<body>
<p>
Your BSU user id is unknown to LON-CAPA, perhaps you don't have a
course in LON-CAPA?
</p>
</body>
</html>
In place of using "sso_failed_login.html", it is also possible to
configure LON-CAPA to allow users from your domain who successfully
authenticate via SSO, but who do not currently have a LON-CAPA
account, to choose to create one. In such cases user information
(first name, last name etc.) can be retrieved from your institution's
LDAP service.
To enable this requires some customization of the standard subroutine
stubs in localenroll.pm and also for a Domain Coordinator to set the
configuration appropriately via:
Main Menu -> Set domain configuration -> Display (User creation and
User modification checked).
The Domain Coordination manual --
http://lon-capa.bsu.edu/adm/help/domain.manual.pdf -- (section 2.7
"Identity Management: Creating New Users") discusses how to allow
users authenticated by institutional log-in to create their own new
LON-CAPA accounts.
Note: for Shibboleth SSO to work with LON-CAPA, lonshibauth.pm and
lonshibacc.pm need to have been installed in
/home/httpd/lib/perl/Apache and also /home/httpd/html/adm/sso needs to
have been installed.
(The ./UPDATE script used to update to LON-CAPA 2.11.0 will do this for you).
(c) Auto-enrollment
> I have not yet found anything that looks remotely like
> "43551dedcd43febmsul1" among the settings that I can view or modify.
A string such as "43551dedcd43febmsul1" is an internal LON-CAPA
identifier for each course, which will be unique within a domain. As a
domain coordinator, if you click on "More" after the course name at
the top of a LON-CAPA page, after selecting an ad hoc role in a
course, you will see the courseID, of the form:
$domain_$coursenum.
You do not have to generate these IDs (LON-CAPA does this when a
course is created). Note the last part of a course number (msul1 in
the example) is the lonHostID of the server running the session where
the course was created.
What is important however is that LON-CAPA needs to be able to map
your institutional code for the course to the internal course
identifier.
The script used for nightly enrollment updates is:
/home/httpd/perl/Autoenroll.pl (there is an entry in
/etc/cron.d/loncapa).
If you look at that script you will see that for each domain hosted on
a library server, the %courses hash is populated via:
my %courses =
&Apache::lonnet::courseiddump($dom,'.',1,'.','.','.',1,\@hostids,'Course');
The keys in this hash are the courseIDs. %courses is actually a hash
of a hash.
One of the keys in each inner hash is: inst_code (the institutional
code) which you assign when creating the course.
> ... Does the course "name" need to be free of spaces and underscores?
I'm not sure what you mean by course "name". For the course
description (i.e., the course title shown at the top of a page in the
course, and also shown in the course catalog), spaces and underscores
are perfectly acceptable.
> For the "institutionalcode", is there anything I need to know?
My recommendation is to choose a scheme that will allow you to:
1. Separate the components into (for example) Year, Semester,
Department, Course number etc., so you can easily customize
localenroll.pm to auto-catalog official BSU LON-CAPA courses. (e.g.,
see: https://loncapa.msu.edu/adm/coursecatalog)
2. Allow mapping to institutional enrollment data, e.g., via a SQL
query, for a particular course.
The institutional code can include separators such as underscores etc.
You will include perl to split this code into its components within
your customized routines in localenroll.pm.
Once you decide on a format you will customize the various subroutines:
fetch_enrollment(), instcode_format(), and also (optionally)
get_sections(), new_course(), validate_courseID().
> BTW, what would the <authtype> look like if we use Shibboleth? The
> example in localenroll.pm is "<authtype>krb4</authtype>"
If you use Shibboleth SSO, in general the authtype will only be
consulted if users attempt to login to LON-CAPA via the standard
log-in page:
http://<your URL>/adm/login
At MSU we use krb5 because usernames/passwords used for the central
SSO service (a local CAS implementation) are the same as those used
for non-SSO Kerberos service. This allows us to continue to support
LON-CAPA authentication for MSU users via /adm/login at times when SSO
service is unavailable.
If your institution provides an alternate to Shibboleth SSO, at times
when SSO is offline, using the same usernames and passwords, that
would likely dictate what you enter for the default authtype.
You could use: localauth
That would mean that you would customize:
/home/httpd/lib/perl/localauth.pm
to handle authentication. Note: under this scenario, the username and
password will be retrieved from the LON-CAPA log-in page: /adm/login,
so if you are going to support that you might also consider using SSL
on your LON-CAPA server.
In any event once you have your server set up as a Shibboleth SP, the
standard log-in page:
http://<your URL>/
will redirect a user to the authentication page on your institution's
IdP server. After successful log-in the user will be redirected back
to LON-CAPA. At that point either a LON-CAPA session will be
initiated (if there is an account for that username in LON-CAPA) or
the user will be prompted to confirm he/she wishes to create an
account (if you have enabled account creation for SSO users via the
Domain Configuration).
The /adm/login URL will also be the one guest users in your domain
(who do not have central BSU usernames) should use to log-in.
Typically the authtype for those "non-BSU" users would be internal.
The /adm/login URL also supports log-in for users from other (non-bsu)
domains in the LON-CAPA network.
> On the same page, it also says "In order for Autoupdate to work, the
> &allusers_info() routine in localenroll.pm needs to be customized
> and a conduit established to institutional data." I do not find any
> allusers_info routine in localenroll.pm. Do I need to create one?
As noted in the release notes for 2.10.0
(http://mail.lon-capa.org/pipermail/lon-capa-announce/2011/000076.html), and
also in the Domain Coordination manual:
"Note about /home/httpd/lib/perl/localenroll.pm
When you use UPDATE to update an existing LON-CAPA installation to a newer
version, the customizable localenroll.pm file is not overwritten. This
is the file which must be customized to support integration of LON-CAPA
with institutional data sources (e.g., for automated update of course
rosters or user information). Whenever new routines are included in
localenroll.pm these will appear in localenroll-std.pm, which is updated
when a new version is installed.
If you have previously customized localenroll.pm it is recommended that
you compare the contents of localenroll.pm and localenroll-std.pm after
an update to see if there are new subroutines (which exist as stubs in
localenroll-std.pm) which can be copied to your custom localenroll.pm
and later customized, should you wish to use that functionality."
If you do not find an allusers_info() stub in your
/home/httpd/lib/perl/localenroll.pm it likely means that the routine
has been added since localenroll.pm was added to the filesystem on
your server. If so, copy the stub routine from
/home/httpd/lib/perl/localenroll-std.pm and then customize, as needed.
> * a scheduled process on lon-capa.bsu.edu gets list of courses and
> sends to Int
> * Int reads course list and creates:
> - list of sections (for the get_sections sub)
> - list of students in course
> - drop data?
> * fetch_enrollment()
> - gets data files from Int
> - creates classlist files
> - returns file specs and class sizes back to whatever calls it
>
> Does that sound like a sensible arrangement?
Int needs to retrieve a list of students in a course (and their
section affiliation). Int could also retrieve a list of institutional
sections for each institutional course code. drop data are not
needed. LON-CAPA will internally compare the LON-CAPA course roster
with the Institutional roster and drop students identified as
"auto-enrolled" in a LON-CAPA course, who are no longer in the
Institutional roster, if auto-drops are enabled.
Your scheduled process can retrieve LON-CAPA courseIDs for courses in
your domain via:
my %courses =
&Apache::lonnet::courseiddump($dom,'.',1,'.','.','.',1,\@hostids,'Course');
and then can iterate over the outer hash keys, to retrieve a list of
institutional codes, and institutional sections (from the inner hash)
for courses where auto-enrollment is enabled. The process can then
store enrollment data in some way (e.g., flat files, MySQL database
table) on the server. You will then customize
localenroll::fetch_enrollment() to retrieve data from those flat files
or DB table.
localenroll::fetch_enrollment() needs to write the temporary XML files
in:
/home/httpd/perl/tmp/bsu_<cnum>_<courseseccode>_classlist.xml
where <cnum> is the LON-CAPA internal course identifier (e.g.,
43551dedcd43febbsul1), and <courseseccode> is the institutional
identifier for the course section (e.g., fs03nop590001).
> ... What does call fetch_enrollment, BTW? I will have to look at
> that to figure out what courses exist and need to be updated.
localenroll::fetch_enrollment() is called by:
Apache::lonnet::fetch_enrollment_query()
which in turn causes /home/httpd/perl/lonsql to call
localenroll::fetch_enrollment()
There are two circumstance in which
Apache::lonnet::fetch_enrollment_query() might be called:
1. when a Course Coordinator uses:
Main Menu -> Manage course users -> Automated Enrollment -> "Update
roster now"
to update a course roster to the latest institutional data
or
2. when Autoenroll.pm is run from cron -- see entry in: /etc/cron.d/loncapa
Real time roster updates via 1. are really only useful in cases where
an institution permits direct, real time, queries of its institutional
classlist data. In such cases (a) can be used by faculty to update
rosters between nightly runs of Autoenroll.pl
> ... Does allusers_info run during fetch_enrollment?
No.
> Otherwise something (fetch_enrollment?) deletes the *classlist.xml files.
Yes, the temporary files created by fetch_enrollment are unlinked by
the parse_classlist() routine in
/home/httpd/lib/perl/LONCAPA/Enrollment.pm once the data have been
written to perl data structures.
Your customized localenroll::fetch_enrollment() will retrieve
institutional data from somewhere (either queries directly to an
institutional database, or retrieval from a data source populated
periodically by your Int process).
(d) Auto-updates of user information.
&allusers_info() is run by Autoupdate.pl which is a script run nightly
by cron (see entry in /etc/cron.d/loncapa). A domain coordinator will
configure what changes Autoupdate.pl may make via:
Main Menu -> Set domain configuration -> Display (Auto-update settings
checked).
> The manual (p.9) lists several settings for auto_update. Where are
> those settings defined?
Section 2 of the Domain Coordination manual ("Domain Configuration")
describes the various settings which can be made by a Domain
Coordinator via the "Set domain configuration" screens.
If you contact me offlist I can send you the customized localenroll.pm
used at MSU.
Stuart Raeburn
LON-CAPA Academic Consortium
Quoting "Neubauer, Paul" <pneubauer at bsu.edu>:
> Hi Stuart,
>
> Thank you for your excellent answers to my questions.
>
> So, to summarize, it appears that your current recommendation is (at
> least for cases (like ours) where we want to create a relatively
> small and not completely predictable set of courses for inclusion in
> LON-CAPA) that it is probably best to create the courses
> interactively within LON-CAPA rather than trying to generate XML
> files somewhere else for batch creation of the courses. Is that right?
>
> We have not previously been using an autoenrollment procedure, but
> we want to move to doing that. We have also determined that we want
> to authenticate using Shibboleth (which we are currently using for
> other purposes anyway).
>
> I'd like to try to restate what I think we need to do so you can
> tell how well or poorly I understand the problems.
>
> For the Shibboleth part, do we modify localauth.pm? The Domain
> Coordination Tutorial And Manual at
> http://lon-capa.bsu.edu/adm/help/domain.manual.pdf has an example
> for LDAP that appears to take a username and password as parameters
> and then requests authentication from the LDAP server. However as I
> (poorly) understand it, Shibboleth has a different model where the
> user is redirected to the Shibboleth ID server where authentication
> is performed, so that lon-capa never even sees the password, and
> which then hands off suitable authentication tokens to the LON-CAPA
> server. If I more or less failed to garble that too badly, is
> localauth.pm the proper place to make the mods? If so, what do the
> mods look like? Also, what tokens does the Shibboleth server need to
> return for a successful (or unsuccessful) login?
>
> Regarding autoenrollment, I'll try to refer to the example in
> localenroll.pm for my questions.
>
> First: fetch_enrollment requires three arguments. The domain appears
> to be obvious: "msu" for Michigan State courses, "bsu" for Ball
> State courses. The "course" is less obvious. The example has COURSE
> = 43551dedcd43febmsul1 and INSTITUTIONALCODE = fs03nop590001. I am
> guessing that 43551dedcd43febmsul1 is a key that is internal to the
> specific LON-CAPA installation whereas fs03nop590001 is a course
> code that keys into larger MSU data outside of LON-CAPA. Is this
> correct?
>
> I'm guessing that the last 5 characters "msul1" is the name of the
> MSU library server. But what is the rest and where do I find it? Is
> this a standard kind of name that is generated automatically when a
> course is created or is it simply the name entered by the person
> creating the course? For example, when I look at a list of courses
> here, I find a table listing:
>
> Select Course Title Domain Course Code Owner/Co-owner(s)
>
> and when I select a course, I get a page that says something like:
> View/Modify settings for: ASTRO 120 Spring Semester 2012
> I have not yet found anything that looks remotely like
> "43551dedcd43febmsul1" among the settings that I can view or modify.
> Does this imply that we should not be naming courses things like
> "ASTRO 120 Spring Semester 2012" or is the 43551dedcd43febmsul1
> something else (and where would I find it)? Does the course "name"
> need to be free of spaces and underscores?
>
> For the "institutionalcode", is there anything I need to know? It
> appears that MSU uses a nice, predictable, descriptive code with no
> internal "_"s or spaces, but I'm not sure that we have anything
> quite so convenient. We could generate something similar, I suppose
> by parsing a string like fs03nop590001 and using the result in a
> database query, or sh/could we just use a database key like
> "327652"? Our actual LON-CAPA users wouldn't have a convenient way
> of discovering that kind of course key, but I suppose I could supply
> it. I imagine the alternative would be to require course requesters
> to assign something suitable (like your format) in the "Course
> Code" box.
>
> Most likely, policy from above will be that we not be able to
> connect directly to the database and will have to communicate with
> it by means of sftp to put lists of courses and get lists of
> students. I forsee a likely scenario where we have an intermediate
> system (call it Int) where database queries can be run on a
> schedule, so the sequence might be something like:
> * a scheduled process on lon-capa.bsu.edu gets list of courses and
> sends to Int
> * Int reads course list and creates:
> - list of sections (for the get_sections sub)
> - list of students in course
> - drop data?
> * fetch_enrollment()
> - gets data files from Int
> - creates classlist files
> - returns file specs and class sizes back to whatever calls it
>
> Does that sound like a sensible arrangement? What does call
> fetch_enrollment, BTW? I will have to look at that to figure out
> what courses exist and need to be updated.
>
> On the subject of updates, the manual (p.13) says
> "The auto-enrollment process will update user information (name,
> e-mail address, studentID etc.) for any student who is being added
> to the course, but will not change it in other case (i.e., drops,
> section switches) should there be a difference between the values in
> the institutional roster and the values in the LON-CAPA classlist.
> To change user information for students already in the course, the
> Auto-update process must be run"
>
> The manual (p.9) lists several settings for auto_update. Where are
> those settings defined?
>
> On the same page, it also says "In order for Autoupdate to work, the
> &allusers_info() routine in localenroll.pm needs to be customized
> and a conduit established to institutional data." I do not find any
> allusers_info routine in localenroll.pm. Do I need to create one? Is
> there any documentation on what it needs to do, comparable to what
> fetch_enrollment does? Does allusers_info run during
> fetch_enrollment? Otherwise something (fetch_enrollment?) deletes
> the *classlist.xml files.
>
> BTW, what would the <authtype> look like if we use Shibboleth? The
> example in localenroll.pm is "<authtype>krb4</authtype>"
>
> I'm sure I will have other questions along the way, and I suppose I
> could ask them off-list, but I thought it might be a good idea to
> have a reasonable portion of the discussion on-list so that other
> people who may have similar questions in the future might be able to
> find the answers.
>
> Thanks,
> Paul
More information about the LON-CAPA-admin
mailing list