[LON-CAPA-cvs] cvs: modules /bisitz/doc LON-CAPA-Development_Getting-Started-Guide.latex

[bisitz]

bisitz		Thu Jan 12 18:55:57 2012 EDT

  Added files:                 
    /modules/bisitz/doc	
                       	LON-CAPA-Development_Getting-Started-Guide.latex 
  Log:
  Getting Started Guide for new LON-CAPA developers
  
  This is a collection of helpful information which could be found at several places on the internet, mailing lists, etc.
  
  It gives an overview, explains some Linux basics, describes how to setup a development environment and a LON-CAPA server.
  
  
-------------- next part --------------

Index: modules/bisitz/doc/LON-CAPA-Development_Getting-Started-Guide.latex
+++ modules/bisitz/doc/LON-CAPA-Development_Getting-Started-Guide.latex
\documentclass[a4paper]{article}

\usepackage[a4paper, left=2.5 cm, right=2 cm, top=2.5 cm, bottom=3 cm]{geometry}
\usepackage[utf8]{inputenc}\usepackage{latexsym}


\title{LON-CAPA Development - Getting Started Guide}
\author{Stefan Bisitz}
\date{12.01.2012}



\begin{document}

\maketitle


\abstract
Documentation on how to create and configure a LON-CAPA development server including access to and synchronization with the official cvs server hosting all versions of all scripts. A typical use case explains how to use this environment.


\tableofcontents


\newpage



\section{Overview}

\subsection{Prerequisites}
\begin{itemize}
\item Basic Linux knowledge (terminal/command line, basic commands, e.g. ls, cd, cp)
\item Access to a dedicated server (hardware or virtual machine (VM)) with root access
\item High speed internet access recommended
\end{itemize}


\subsection{Components}
\subsubsection{Your local LON-CAPA development server (``your dev server")}
Your local installation of the LON-CAPA server software is located on this machine. You could use dedicated (no other services/programs besides LON-CAPA are running) hardware or a virtual machine. Your scripts will run on this server including testing and debugging.


\subsubsection{Local cvs workspace (``your workspace")}
Your worksapce contains a copy of the current or your special versions of all LON-CAPA scripts. That's the place where you change the scripts and connect to the cvs server (see below).


\subsubsection{LON-CAPA cvs server (``cvs server")}
The main LON-CAPA developers at Michigan State University (msu) host a server with a versioning tool called cvs (concurrent Versions System). This special server provides all versions of all LON-CAPA scripts. This includes the first version ever made of a script, all intermediate versions, the latest version and some special versions, e.g. production release versions or development debugging versions.

You can retrieve the desired script versions for your local development purposes from this server as well as you send your script changes from your workspace to this server to make them available to the other developers and offer them as a new official development version.


\subsection{Location}
It's not a must to have your cvs workspace on the same server on which your LON-CAPA development server runs. However, this brings some advantages. The following instructions expect to use the same server for both.



\section{Creating and configuring ``your dev server"}

\subsection{Install Linux Operating System (OS)}
\begin{itemize}
\item Use a dedicated server, e.g. a VM
\item Install OS with base system only, e.g. CentOS 5.7 (minimal installation)
\end{itemize}
The following examples assume you use CentOS.


\subsection{Install LON-CAPA}
Detailed installation guides can be found at
\begin{verbatim}
http://install.lon-capa.org
\end{verbatim}


\subsection{Switch to development repositories}
Replace the configuration file for OS updates \emph{yum.conf} with the special LON-CAPA developer's version:

(as root:)
\begin{verbatim}
cd /etc/
cp yum.conf yum.conf.backup_prodversion
wget http://install.loncapa.org/developer/centos/5/centos5_dev_loncapa_yum.conf
mv centos5_dev_loncapa_yum.conf yum.conf
yum update
\end{verbatim}

This will most likely install some packages needed in the developer's version of LON-CAPA.

Official details:
\begin{verbatim}
http://mail.lon-capa.org/pipermail/lon-capa-dev/2007-November/001852.html
\end{verbatim}



\section{Creating and configuring ``your workspace"}

\subsection{Create a dev user}
Create a local user for development purposes. Replace \emph{mydevuser} with the desired user name and set a secure password. Warning: Do not use a user name which is already used or is planned to be used as a LON-CAPA user name to avoid conflicts.

(as root:)
\begin{verbatim}
useradd mydevuser
passwd mydevuser
\end{verbatim}


\subsection{Install cvs}
Install cvs on your local machine with your workspace unless it is already installed.

(as root:)
\begin{verbatim}
yum install cvs
\end{verbatim}


\subsection{Create rsa key files for authentication with cvs server}
\label{creatersakey}
To get read/write access to the cvs server, a developer needs an account on the cvs server. To be able to authenticate, you need a rsa key. If you don't already have one, you need to create a pair of private and public key:

Login on your dev server with your local dev user or use another Linux system where ssh-keygen is available.
\begin{verbatim}
ssh-keygen -t dsa
\end{verbatim}

If you choose a passphrase, you will always be asked to enter it when proccessing any cvs command. This is a plus in security but could also be annoying. Don't forget to remember the passphrase. If you forgot it, a new key would be needed (and sent to the cvs server team to replace it there). If you cannot be sure that nobody else has access to your computer, you should use a passphrase for security reasons. Be aware of that everybody who has access to your computer can process cvs commands without additional authentication!

Transfer the key files to your system by using, e.g.
\begin{itemize}
\item \emph{sftp} (Linux)
\item \emph{WinSCP} (Windows)
\end{itemize}

Protect your private key and keep it save! Back it up on a secure medium! If you lost your private key, you could not communicate with the cvs server anymore. If the file was lost forever, you would need to create a new one and send the new corresponding public key to the cvs server team again.

Official documentation about how to create the files:

\begin{verbatim}
http://help.loncapa.org/cgi-bin/fom?_recurse=1&file=4#file_18
\end{verbatim}
Section: ``Using a public key"

\subsection{Request personal cvs account}
\label{requestcvsaccount}
Ask your local development team to request an account for you. You probably want to use the same account name as your institutional account. Your cvs account will be used later on to allow you to connect to the cvs server.

Your public key file \emph{id\_dsa.pub} (\ensuremath{\rhd} \ref{creatersakey}) needs to be attached to the request. In contrast, keep your private key secret!

You most likely need read and write access to the \emph{loncapa} folder and sub folders. The request will be sent to \emph{helpdesk at lon-capa.org} and usally takes only a few days to be processed.

Wait for your cvs account to be created. Once you get the confirmation, you can continue with the steps as follows.




\section{Configuring connection to ``cvs server"}
The following steps won't work, if your cvs account hasn't been created yet.

The cvs server is already installed and properly running. All you need to do is to configure your local workspace so that you get access from your local workspace to the cvs server. This allows you to send data to and retrieve data from it.

\subsection{Place your private key file}
\label{placekey}
Use your local dev user for the upcoming commands to apply appropriate file privilegs. If you don't, you will need to change the privs accordingly afterwards.

Copy your private key file \emph{id\_dsa} to your dev server to the folder: \emph{/home/yourlocaldevuser/.ssh/}

Note: Files and folders which start with a ``." are hidden by default. To display them, you need to explicitly show them, e.g. \emph{ls -a}

\subsection{Make cvs variables available to local system/terminal}
\label{cvsvars}
Add the following two lines to the \emph{.bashrc} file of your dev user (\emph{\~{}/.bashrc}) and save it. Replace \emph{USERNAME} with your actual cvs user account. Make sure that you use your username of your personal cvs account as requested earlier (\ensuremath{\rhd} \ref{requestcvsaccount}), not the local dev user.
\begin{verbatim}
export CVS_RSH=ssh
export CVSROOT=:ext:USERNAME at source.lon-capa.org:/home/cvs
\end{verbatim}

The export command sets the environment variables needed for CVS access. Storing them in your dev user's \emph{.bashrc} file will ensure that the variables are automatically set each time you login with your local dev user.

Immidiately activate variables:
\begin{verbatim}
source ~/.bashrc
\end{verbatim}

You can check if the variables are properly set by showning their content with the commands:
\begin{verbatim}
echo $CVS_RSH
echo $CVSROOT
\end{verbatim}


\subsection{Create a folder for your cvs workspace}
Login on your dev server with your dev user. Choose an appropriate place and name, e.g. \emph{\~{}/cvs-loncapa/} (assumed to be used in the following examples).
\begin{verbatim}
cd ~
mkdir cvs-loncapa
\end{verbatim}


\subsection{Initially retrieve the current development versions of all scripts}
(as local dev user):
\begin{verbatim}
cd cvs-loncapa
cvs checkout loncapa
\end{verbatim}
Doing this the first time will prompt to confirm the RSA key fingerprint.

If you are prompted to enter a password (password, not passphrase), the usual issue is that your key file is not accessible (\ensuremath{\rhd} \ref{placekey}), the environment variables are not set properly (\ensuremath{\rhd} \ref{cvsvars}) or the account is not correctly configured on the cvs server (\ensuremath{\rhd} \ref{requestcvsaccount}).


\subsection{Configure hosts and domain files}
\label{hostsdomainfiles}
There are four files related to the LON-CAPA hosts and domains:
\begin{verbatim}
domain.tab
dns_domain.tab
hosts.tab
dns_hosts.tab
\end{verbatim}

To avoid error messages due to missing links when processing an installation process, you need to set soft links to the appropriate hosts and domain files. This seems always a bit tricky to set the correct links and depends on several other configurations. Try this:
(as local dev user):
\begin{verbatim}
cd ~/cvs-loncapa/loncapa/loncom/
ln -s development_domain.tab domain.tab
ln -s development_hosts.tab hosts.tab
\end{verbatim}

Important notes:
\begin{itemize}
\item If your domain is not part of the official development cluster, you need to add your server explicitly to the files \emph{domain.tab} and \emph{hosts.tab}.
\item If your server has an internal hostname but no DNS entry in the network, you need to replace the hostname in the row related to your server in the file \emph{hosts.tab} with the IP address of your server. Use this IP address to access the server via the web browser.
\end{itemize}

\section{Typical Use Case}
Typical use case about how to change a LON-CAPA script, test it and eventually update the cvs server with your local changes.


\subsection{Update/synchronize your workspace}
Update your workspace with the desired script versions. You likely need the most recent versions of all scripts:
\begin{verbatim}
cd ~/cvs-loncapa/loncapa/
cvs update -d -A
\end{verbatim}

Important parameters for \emph{cvs update}:
\begin{description}
\item[-d]include sub folders
\item[-A]remove all sticky tags and ensure usage of latest version
\item[-r]explicit version number or (release) tag
\end{description}

Examples:
\begin{description}
\item[cvs update -d -A]Update all scripts in the current folder and all sub folders and synchronize with the latest script versions.

\item[cvs update -d -r\texttt{"}version\_2\_10\_1\texttt{"}]Update all scripts in the current folder and all sub folders and synchronize with the script versions assigned to the release tag \emph{version\_2\_10\_1}

\item[cvs update -r1.205 loncommon.pm]Update the script loncommon.pm in the current folder and synchronize with the script version 1.205
\end{description}


\subsection{Locate scripts to be changed}
\begin{itemize}
\item Login with your dev user on your dev server
\item Go to the workspace folder
\begin{verbatim}
cd ~/cvs-loncapa/loncapa/
\end{verbatim}
\end{itemize}

You need to find the scripts which are involved. There is no general rule to find them. One approach, depending on what kind of thing you search for, could be to find a phrase in a script which appears on the screen in the web browser.

Helpful commands:
\begin{itemize}
\item grep
\item locate
\item Texteditor, e.g. vi, vim, nano
\end{itemize}


\subsection{Change scripts}
Use your preferred text editor to change the scripts. Only apply changes directly in the scripts within your workspace folder. Never ever copy a script from outside into the workspace! cvs wouldn't recognize the change and could behave odd from your point of view. If you really need to copy a file from outside the workspace back into it, ask a cvs expert how to do this properly.


\subsection{Apply your script changes to the dev server}
\label{applyscriptchanges}
Copy the changed scripts from your workspace to the appropriate folder on your dev server. Most scripts are located in the Apache folder of your dev server. A few are located somewhere else. If unsure, find the correct location, e.g. use \emph{locate scriptname}. Watch out that there might be an installation folder where you would also find a copy of the scripts. Do not touch them.

\paragraph{a) Send single file from your workspace to your dev server}
For easier recovery of the old version, you might first want to make a backup of the file you are going to overwrite:

(as root:)
\begin{verbatim}
cd /home/httpd/lib/perl/Apache/
cp scriptname scriptname.backup
\end{verbatim}

Copy a single file from your workspace to your dev server (as root). \emph{yourdevuser} needs to be replaced with your actual dev user.
\begin{verbatim}
cp /home/yourdevuser/cvs-loncapa/.../yourchangedscript /home/httpd/lib/perl/Apache/.
\end{verbatim}

\paragraph{b) Full installation}
\label{fullinst}
If more than a few scripts are changed, major changes have occurred or a full version update is desired, you run a full installation instead of copying all scripts one by one.

(as dev user:)
\begin{verbatim}
cd ~/cvs-loncapa/loncapa/loncom/build/
make clean
make build
su
make install
/etc/init.d/loncontrol stop
/etc/init.d/loncontrol start
/etc/init.d/httpd restart 
\end{verbatim}

This only works if your workspace is located on the same machine. If not, copy your workspace folder to the destination machine before executing the steps above. It could be needed to adjust the hosts and domain files (\ensuremath{\rhd} \ref{hostsdomainfiles}).

You could leave out \emph{make clean} and \emph{make build} in many cases. General rule: If \emph{make install} failed, try again with the two other commands.

Check out the \emph{WARNINGS} files for notes, warnings and errors after each make command. These files will be overwritten in the next installation run. It's up to you to keep a backup.


\subsection{Activate the changes on your dev server}
Unless you haven't already done (\ensuremath{\rhd} \ref{fullinst}), in most cases it is enough to reload the web server to activate the changes you've applied:
\begin{verbatim}
/etc/init.d/httpd reload
\end{verbatim}
However, changes to a few scripts need a restart of \emph{loncontrol} to get them activated:
\begin{verbatim}
/etc/init.d/loncontrol restart
\end{verbatim}


\subsection{Track errors in the changed scripts}
Your changes to a script might result in an incorrect syntax or other unwanted behaviour which leads to a dysfunctional server. This needs to be corrected - obviously.

Watch out that an incorrect script would not prevent the web server from reporting \emph{[ok]} if you have applied a simple \emph{httpd reload}. Anyway, you could us it, because a restart would be slower than a reload. Once you call the related web page or even any page in LON-CAPA, depending on the error, you will get an error message in the browser window.

\begin{enumerate}
\item Error log

Check the error log for helpful advice:
\begin{verbatim}
/var/log/httpd/error_log
\end{verbatim}
The latest entries are at the bottom of the file. The first line of the last block is where you likely want to search for the actual error.
\item Correct your script

Change your incorrect script in your workspace.
\item Copy the script again to the server (\ensuremath{\rhd} \ref{applyscriptchanges})
\item Start the web server

After the web server encountered an error and you have corrected the erronous script, you need to (re)start the web server:
\begin{verbatim}
/etc/init.d/httpd start
\end{verbatim}
\end{enumerate}

\subsection{Test your changes}
Always test your changes! You can never be sure if everything works as expected. Make sure the desired behaviour is achieved and don't forget to test all special cases, e.g. parameter is empty, user data has wrong format, etc.

Watch out for side effects! Make sure your changes don't break things at other places, e.g. changing the signature of a function implies that you change \emph{every} single call of this function in the \emph{whole} system; changing a phrase could mean that another place where it is also used doesn't make sense anymore, etc.


\subsection{Send your improved scripts to the cvs server}
\begin{enumerate}
\item Make sure again that the (web) server still runs with the changed scripts
\item Check which scripts you have changed
\item Check which changes you actually applied
cvs diff scriptname
This compares your local version with the latest version on the cvs server. Use \emph{cvs diff -rV1 scriptname} to compare your local version with version V1 instead of the latest version. Use \emph{cvs diff -rV1 -rV2 scriptname} to compare version V1 with version V2 ignoring your local version.
Common mistake is to forget to remove debugging code. The \emph{cvs diff} would reveal this.
\item Prepare log entry

Prepare the log entry to explain briefly but exactly what you have changed and why. Write a little summary of your changes. Do not copy the code or diff results. That's available anyway. Better try to convince the other developers why your change was needed and what was needed to achieve this.
\item Send your changes to the cvs server

Have your tests been successful? Are all other functions still available? If you cannot answer all these questions with YES, fix these issues before continuing.
\item \emph{cvs commit scriptname}

This will prompt you to enter the log entry. The first time you commit, you need to specify your preferred text editor to edit your log entries.

After having saved and exited the editor, your changes and the log entry will be sent to the cvs server. Wait for confirmation. A new version number will be created and shown to you on the screen. Make sure that this has been processed properly!
\end{enumerate}
And finally... Be happy about your contribution to this great Open Source project!


\subsection{Important notes}
\begin{itemize}
\item Only one version of a script is available in one workspace at the same time. However, you could copy a script with the desired version to another place. This should be outside the workspace folder.
\item To commit to the main developer branch, only changes to the latest version of a script are possible. A commit of another version would be cancelled with a warning. You first needed to update to the latest version.
\item If you have changed a script and then run a \emph{cvs update}, your changes are automatically merged into the requested version. This might conflict. It is highly recommended to correct the conflicting parts before you change anything else.
\end{itemize}

\end{document}