Difference between revisions of "Software Development (WG1) Version Control (T3)"

Revision as of 16:18, 7 March 2007

Setting up a VistA Development Environment

This page uses the historical meaning of the term "OpenVistA" VistA Trademark Issues

Overview

This is a description of how VistA development environments should be configured.

Monospaced text designates program names, directory names, command line parameters, etc., that are intended to be typed in at a Linux shell or a GT.M prompt, e.g. /bin/bash. Italic text is a descriptor of something whose actual value must be determined when the command is typed, e.g., gtmver.

Theory

A release is a collection of routines and global variables. The canonical packaging of a release, which is suitable for importation into any standard M distribution, is a collection of M source code routines and a database extract in ZWRite format. When installed on a system, to run with an M implementation, a release may have pre-compiled object files, database files, shell scripts, global directories, etc.

Below is a diagram of the “roll-up” by which a VistA release (“Release A”) gets converted into another VistA release (“Release B”) by a development team. A and B could be from different families, e.g., Release A could be a FOIA release and Release B could be a Leonardo da VistA release.

Once Release A is installed, it remains frozen and unchanged except for any essential patches that may become available independent of the development process for creating Release B and without which the release lacks some required functionality. This means, for example, that KIDS patches should be applied in an integration environment rather than a release environment.

At any given time, there can be multiple development projects underway based on Release A. In other words, there may well be a Release C that is also based on Release A, but which has its own integration and development areas.

To start the development process, an integration directory is created and initialized with a copy of the global variables from Release A, as well as shell scripts, global directories, etc. The integration directory may well have a different name from Release A and Release B. For example, Release A may be FOIAVistA_20051021, Release B may become LeonardoDaVistA_0.4 (at the time that the integration area is created, the name and version of Release B may not even be decided yet), and the integration and development areas may be named Greenbelt_200510. There is no need to copy routines, since the GT.M search path will be set up to look for routines in the integration area before looking in Release A; hence only routines that are different between Release A and the new release need to be duplicated.

Actual development does not take place in the integration area. Development occurs in development areas, typically in the directories of individual developers; however, there may be a shared development area if two developers are collaborating on making changes to the very same module (i.e., their code changes overlap). A development area is set up by creating a directory structure, shell scripts, etc. Note that development areas can be hierarchical so that development subtasks 1a and 1b can be developed separately and integrated into development task 1, prior to integration into the integration area, etc.

During this development process, developers develop code in individual sand boxes (development areas), but normally share the database (global variables) in the integration area – there may typically be no need to create a separate database. However, if some global variable changes are tied to code changes being made in a development area, a copy of those global variables can be placed in a separate database file in that development area, and a global directory for that development area can map those global variables to the database in that development area, with all other global variables mapped to the database in the integration area.

When a developer has code that is ready for integration, the new versions of the modified routines are promoted to the integration area. Any global variables tied to the code being promoted that are in a database in the development area must be merged, reconciled or otherwise integrated with the global variables in the database in the integration area and the global directory in the development area modified accordingly. Note that there needs to be a process, e.g., involving code reviews, by which a development task is considered complete enough to be promoted. These, and other quality gates, are not discussed herein.

Creating a release is minimally the process of creating the directory structure for Release B, copying the routines from Release A, overlying the routines with those from the integration area, and copying the global variables from the integration area.

Normally, Release A, the integration area and development areas would all use the same GT.M version. However, this is not a requirement.

It is strongly recommended that database files in integration and development environments be journaled, and backed up regularly. It is also strongly recommended that a version control system such as CVS or subversion be used for integration areas.

Practice

Shell

The standard shell for VistA community scripting is bash (installed as /bin/bash on Debian GNU/Linux systems).

GT.M Release Directories

Each release of GT.M is installed its own subdirectory of /opt (preferred) or /usr/local (acceptable). Each directory name starts with gtm followed by a release number separated from the release name by an underscore (_). Thus, for example, GT.M V5.0-000C would be installed in /opt/gtm_V5.0-000C. /opt/gtm is a relative symbolic link to the latest version, set up with a command sequence such as:

 cd /opt
 rm -f gtm
 ln -s gtm_V5.0-000C gtm

When installing GT.M, the file gtmprofile should be edited so that the line:

 EDITOR="/bin/vi"; export EDITOR

is modified to read:

 export EDITOR=${EDITOR:=/bin/vi}

VistA Release Directories

Each release is installed in its own sub-directory of a standard system location, such as /opt (preferred) or /usr/local (acceptable). Each directory has a release family name followed by a release / version number suffix separated from the release name by an underscore. Directories for releases without version numbers, such as FOIA releases, have a date suffix of the form yyyymmdd (and where appropriate just yyyymm). For example, an August 25, 2005 FOIA release might be in /opt/FOIAVistA20050825, an October 21, 2005 release in /opt/FOIAVistA20051021, an OpenVistA 0.3 release in /opt/OpenVistA0.3 and a Leonardo da VistA 0.45 release in /opt/LeonardoDaVistA0.45.

The normal protection on all directories and files in a release directory should be read-only. To install files in the p subdirectory, it should be made read-write, the source file(s) for the patch copied in, the object file(s) generated, and the directory and its contents again made read-only.

For each release family, the release directory contains subdirectories g, o, r and p, symbolic link gtm, a script install, scripts run, and cprs_direct that are not executable in a release directory, but which are used when copied to integration directories and made executable. A file env is used to set up the execution environment.

Other files and shell scripts may be created for various maintenance purposes (and ideally would be documented here).

• Subdirectory g contains a global directory file mumps.gld and a compressed database file mumps.dat.gz. The global directory maps all global variables to a single database file $vista_home/g/mumps.dat (i.e., the environment variable $vista_home is used at run-time to point a GT.M process to the location of the database file, so that different processes can actually use the same global directory to refer to entirely different database files).

• Subdirectories o and r correspond to directories for files containing routine object code and routine source code respectively. Each file in r ends in .m and has a corresponding file in o with a newer time stamp and which ends in .o.

• Subdirectory p contains source and object code for patches to a release (generated outside the scope of the work for creating Release B) and which are deemed to be essential for the proper functioning of the release.

• The symbolic link gtm points to the GT.M directory used for this VistA directory.

• The file env sets up the GT.M environment:

# env - file to be sourced to create VistA environment
#
# This temporary version of the commands to set up the VistA
# environment assumes that the parent and child use the same
# version of GT.M.

if [[ -d parent ]] ; then
  pushd parent 1>/dev/null
  source ./env
  popd 1>/dev/null
fi

if [[ -n $routines ]] ; then
  export routines="$PWD/p $PWD/o($PWD/r) $routines"
else
  export routines="$PWD/p $PWD/o($PWD/r)"
fi
if [[ -f $PWD/g/mumps.dat ]] ; then export vista_home=$PWD ; fi

source gtm/gtmprofile
export gtmgbldir=$PWD/g/mumps.gld
export gtmroutines="$routines $gtm_dist"

• The script install creates a new integration directory to be used in creating a new release from this release. Normally, the initial database file of the integration directory is created by uncompressing mumps.dat.gz. When a development directory is created as a child of an integration environment, or as a child of another development environment, the command line switch --separate-globals can be used to specify that the child is to have its own global variables. Here is the script install:

#!/bin/bash
#
# install - create new VistA environment as child of an existing environment
#
# Usage:
#
# install [--separate-globals] newdirectory
#
# Limitations:
#
# 1. If the globals in the parent have a custom partitioning or a custom
#    partitioning is desired for the child, don't use this script.
# 2. There is no error handling.  Clean up by deleting the child manually.
# 3. Parent and child must use the same version of GT.M.

TRUE=0                                        # Symbolic constant(s)

if [[ 0 -eq $# ]] ; then echo "Not enough information specified.  Exiting." ; exit 1 ; fi

case $1 in

  --separate-globals)                         # child gets own database
    separate=$TRUE
    child=$2
  ;;

  *) # global variables are shared with parent
    child=$1
  ;;

esac

if [[ -e $child ]] ; then echo $child " exists already.  Exiting." ; exit 1 ; fi

pushd `dirname $0` 1>/dev/null
export parent=$PWD
source ./env
popd 1>/dev/null
echo "Creating environment in " $child " as child of environment in " $parent

# Create the directory structure for the child
mkdir -p $child/{g,o,r,p,tmp}

# Link the child to the parent
ln -s $parent $child/parent

# Give the child the same GT.M version as the parent
ln -s $parent/gtm $child/gtm

# Copy other files in this directory
find $parent/ -maxdepth 1 -type f \! -name \*~ -exec cp {} $child/ \;

# Make run & cprs_direct executable in child; they may not be executable for the parent
chmod u+x $child/run $child/cprs_direct
find $child -maxdepth 1 \( -name run -o -name cprs_direct \) -perm -4 -exec chmod o+x {} \;
find $child -maxdepth 1 \( -name run -o -name cprs_direct \) -perm -40 -exec chmod g+x {} \;

# Copy global directory
cp $parent/g/mumps.gld $child/g/

# Several cases to consider with respect to the database and global directory for the child:
# 1. Parent database is zipped: Child always gets unzipped database from parent
# 2. --separate-globals is specified without any arguments: child gets copy of database from parent
# 3. --separate-globals is not specified: Child shares database with parent; nothing to copy

if [[ -e $parent/g/mumps.dat.gz ]] ; then      # case 1 above
  echo "Unzipping from parent to create initial database"
  gzip -d <$parent/g/mumps.dat.gz >$child/g/mumps.dat

elif [[ $separate ]] ; then                    # case 2 above
  echo "Backing up copy of parent database to child environment"
  $gtm_dist/mupip backup DEFAULT $child/g/

fi

# Set up journaling for child
if [[ -e $child/g/mumps.dat ]] ; then
  $gtm_dist/mupip set -journal="enable,on,before,file=$child/g/mumps.mjl" -file $child/g/mumps.dat
  chmod o+r,o-w,g+w $child/g/mumps.{dat,mjl}
fi

# Make files except database files, journal files & directories read-only
find $child -type f \! -name \*.dat \! -name \*mjl -exec chmod a-w {} \;
find $child -type d -exec chmod 775 {} \;

echo "Default permission for development environment is for all to read and group to write - please alter as needed"

In a future enhancement, option --newgtm gtmver, will specify that the integration directory is to be configured to run a different version of GT.M (as specified by the directory gtmver) from the one used by the release. If --newgtm gtmver is used, the default assumption will be that the database will need to be extracted with the version of GT.M used by Release A, and loaded into a new database created with the version of GT.M to be used in the integration directory. When --newgtm gtmver is used, a an additional optional option --nonewdb will be available to specify that the releases of GT.M have compatible database formats, and that database from Release A can be used unchanged in the integration directory. Note that the use of --newgtm gtmver will almost certainly result in a command that takes a long time, because the database may need to be extracted and reloaded, and the routines recompiled.

Here is a listing of a top level release directory:

bhaskar@Makaira:~$ ls -l /opt/FOIAVistA20051021
total 912
dr-xr-sr-x  2 root staff   4096 Sep 15 13:19 CPRS_Gui
-r-xr-xr-x  1 root staff    216 Nov 22 09:21 cprs_direct
-r--r--r--  1 root staff    583 Nov 22 09:21 env
dr-xr-xr-x  2 root staff   4096 Oct 27 14:00 g
lrwxrwxrwx  1 root staff     18 Nov 22 09:26 gtm -> /opt/gtm_V5.0-000C
-r-xr-xr-x  1 root staff   1832 Nov 22 10:44 install
dr-xr-xr-x  2 root staff 446464 Nov 22 07:10 o
dr-xr-sr-x  2 root staff   4096 Nov 22 09:22 p
dr-xr-xr-x  2 root staff 446464 Oct 27 13:38 r
-r--r--r--  1 root staff    252 Nov 22 09:21 run
-r-xr-xr-x  1 root staff   3633 Oct 30 06:42 vista

VistA Integration Directories

Here is an example of creating an integration environment from a release environment:

vista@Makaira:~$ /opt/FOIAVistA20051021/install LeonardoDaVistA0.1
Parent is /opt/FOIAVistA20051021
Unzipping from parent to create initial database
%GTM-I-JNLCREATE, Journal file /home/vista/LeonardoDaVistA0.1/g/mumps.mjl created for database file /home/vista/LeonardoDaVistA0.1/g/mumps.dat with BEFORE_IMAGES
%GTM-I-JNLSTATE, Journaling state for database file /home/vista/LeonardoDaVistA0.1/g/mumps.dat is now ON
Default permission for development environment is for all to read and group to write - please fix if needed
vista@Makaira:~$

A VistA integration directory is similar to a VistA release directory, with the following differences.

• Subdirectory g contains a database file, mumps.dat, rather than a compressed database file, mumps.dat.gz.

• The symbolic link parent points to the release directory from which this integration directory was created. A release directory does not have a symbolic link called parent. (The recommended way for a script to determine whether a directory is a release directory or an integration / development directory is to test for the existence of parent.)

• A script run which invokes and runs VistA in that integration environment. Note that a VistA release may physically include a run script, but it will not work because it is not possible to directly run VistA from a release directory. Here is the run script:

#!/bin/bash
#
# run - run this instance of VistA

# Set up the environment for VistA
pushd `dirname $0` 1>/dev/null
export parent=$PWD/parent
source ./env
popd 1>/dev/null

if [[ -n $1 ]] ; then
  $gtm_dist/mumps -run $1
else
  $gtm_dist/mumps -dir
fi

• The cprs_direct script handles connection requests from CPRS GUI client processes that come in via inetd/xinetd:

#!/bin/bash
#
# cprs_direct - start a process to serve the CPRS Gui client

# Set up the environment for VistA
export HOME=/home/`whoami`
cd `dirname $0`
export parent=$PWD/parent
source ./env

# Run the server for the CPRS GUI client
cd tmp
$gtm_dist/mumps -run GTMLNX^XWBTCPM

• When the install script is executed, it will create a development directory.

• Except for database files, journal files, and directories, the normal protection on all directories and files in an integration directory should be read-only. When promoting code to the integration directory from a development directory, or to install externally generated patches in the p subdirectory, needed directories should be made read-write, sources copied in, object file(s) generated, and directories again made read-only.

Below is a listing of the directory of an integration environment created from the release in the example above. Notice parent link pointing to the directory of the parent environment. (In this case, the gtm link points to the gtm link in the parent, rather than /opt/gtm_V5.0-000C. It doesn't matter either way.)

bhaskar@Makaira:~$ ls -l /home/vista/LeonardoDaVistA0.1
total 48
-r-xr-xr-x  1 vista vista  216 Nov 22 09:28 cprs_direct
-r--r--r--  1 vista vista  583 Nov 22 09:28 env
drwxrwxr-x  2 vista vista 4096 Nov 22 10:11 g
lrwxrwxrwx  1 vista vista   26 Nov 22 09:28 gtm -> /opt/FOIAVistA20051021/gtm
-r-xr-xr-x  1 vista vista 1832 Nov 22 10:44 install
drwxrwxr-x  2 vista vista 8192 Nov 22 09:33 o
drwxrwxr-x  2 vista vista 4096 Nov 22 09:28 p
lrwxrwxrwx  1 vista vista   22 Nov 22 09:28 parent -> /opt/FOIAVistA20051021
drwxrwxr-x  2 vista vista 8192 Nov 22 09:32 r
-r-xr-xr-x  1 vista vista  252 Nov 22 09:28 run
drwxrwxr-x  2 vista vista 4096 Nov 22 09:28 tmp
-r-xr-xr-x  1 vista vista 3633 Nov 22 09:28 vista

VistA Development Directories

A development directory is like an integration directory, except that it does not normally have its own database file or global directory. Here is an example of creating a development environment from the integration environment:

bhaskar@Makaira:~$ ~vista/LeonardoDaVistA0.1/install LeonardoDaVistA0.1
Parent is /home/vista/LeonardoDaVistA0.1
Using parent database in this environment
Default permission for development environment is for all to read and group to write - please fix if needed
bhaskar@Makaira:~$

Note that there is hardly anything in the directory of a development environment:

bhaskar@Makaira:~$ ls -lR LeonardoDaVistA0.1/
LeonardoDaVistA0.1/:
total 40
-r-xr-xr-x  1 bhaskar vista  216 Nov 22 14:09 cprs_direct
-r--r--r--  1 bhaskar vista  583 Nov 22 14:09 env
drwxrwxr-x  2 bhaskar vista 4096 Nov 22 14:09 g
lrwxrwxrwx  1 bhaskar vista   34 Nov 22 14:09 gtm -> /home/vista/LeonardoDaVistA0.1/gtm
-r-xr-xr-x  1 bhaskar vista 1832 Nov 22 14:09 install
drwxrwxr-x  2 bhaskar vista 4096 Nov 22 14:09 o
drwxrwxr-x  2 bhaskar vista 4096 Nov 22 14:09 p
lrwxrwxrwx  1 bhaskar vista   30 Nov 22 14:09 parent -> /home/vista/LeonardoDaVistA0.1
drwxrwxr-x  2 bhaskar vista 4096 Nov 22 14:09 r
-r-xr-xr-x  1 bhaskar vista  252 Nov 22 14:09 run
drwxrwxr-x  2 bhaskar vista 4096 Nov 22 14:09 tmp
-r-xr-xr-x  1 bhaskar vista 3633 Nov 22 14:09 vista

LeonardoDaVistA0.1/g:
total 4
-r--r--r--  1 bhaskar vista 1024 Nov 22 14:09 mumps.gld

LeonardoDaVistA0.1/o:
total 0

LeonardoDaVistA0.1/p:
total 0

LeonardoDaVistA0.1/r:
total 0

LeonardoDaVistA0.1/tmp:
total 0
bhaskar@Makaira:~$ du -sh LeonardoDaVistA0.1/
48K     LeonardoDaVistA0.1/
bhaskar@Makaira:~$

The ability to create a development environment that can be isolated from the integration environment with an overhead of 48K bytes makes it easy to create many development environments as sand boxes to simplify the development process.

As discussed earlier, it may be entirely appropriate for a development directory to have database files when there is a need to isolate certain globals.

Indeed, since development directories are hierarchical – a development directory is an integration directory for development directories below it, and itself contributes to an integration directory above it, an integration directory is really nothing more than a development directory that does not itself contribute to an integration directory above it, but instead is used to create a release.

Note that although a GT.M database file can be opened by only one GT.M version at a time, it is quite easy to create a directory structure where Release A, the integration area, and each development area use different versions of GT.M. In the most common case, where the integration and development areas use a different version of GT.M from Release A, the only additional configuration requirement is to create a directory in the integration area for object files generated from the source files in Release A with the version of GT.M used for integration and development of Release B