Building Packages with Hudson

General

Hudson is a continuous build environment used by DataONE and is located at:

http://dev-testing.dataone.org:8080/hudson

Debian Packages

Two Debian Apt repositories are managed by Hudson: ubuntu-unstable and ubuntu-stable. The ubuntu-snapshot repository holds Debian packages built from the development trunk.

The ubuntu-unstable repository is located at:

http://dev-testing.dataone.org/ubuntu-unstable

and can be configured in /etc/apt/sources.list by adding an entry:

deb http://dev-testing.dataone.org/ubuntu-unstable lucid universe

The ubuntu-stable repository is located at:

http://dev-testing.dataone.org/ubuntu-stable

and can be configured in /etc/apt/sources.list by adding an entry:

deb http://dev-testing.dataone.org/ubuntu-stable lucid universe

Building Unstable / Snapshot Packages

The unstable or snapshot Debian packages contain code compiled from the development trunk. These packages are rebuilt in response to commits to any source in the hierarchy on which they are dependent, which includes the source of the packages themselves or any of the source contributing to binaries they contain. The general sequence for building these packages is:

1. Checkout latest copy from Subversion trunk
2. Copy the compiled binaries (usually .jar and .war files) into the PACKAGE/usr/share/PACKAGE_DIST folder.
3. Copy everything into a build location, removing any SVN chaff
4. Update the Version information for the package
5. Build the .deb file(s)
6. Copy the .deb(s) to the repository location
7. Update the Packages.gz file for the repository
8. Send out notification to IRC channel #dataone-build

Steps 1 and 8 are managed by the Hudson job, steps 2-7 are performed with a Bash script executed by Hudson, described below.

A generic script that can be modified as necessary for building a Debian package is provided below. Key variables used in the script are listed below. Of these, PACKAGE and JARS must be set, and PACKAGE_DIST should be checked for correctness.

PACKAGE:Name of the resulting Debian package. e.g. d1-cn-os-core PACKAGE_DIST:Name of the folder under usr/share within the Debian package that will contain the compiled binaries. Generally the same value as PACKAGE, but not always. JARS:An array of paths to compiled binaries that are to be added to PACKAGE_DIST. The paths are relative to the Hudson jobs folder, which is $WORKSPACE/../... e.g., for the dataone-cn-index job, JARS = ( “d1_cn_index_tool/workspace/target/d1_index_build_tool.jar” “d1_cn_index_generator/workspace/target/d1_index_task_generator_daemon.jar” “d1_cn_index_processor/workspace/target/d1_index_task_processor_daemon.jar” ) APTREPOS:The absolute file system path to the location of the Apt repository. This value is set by a global environment variable (UBUNTU_UNSTABLE) set in Hudson configuration (Menu: Manage Hudson / Configure System / Global Properties ). WORKSPACE:A value set by Hudson that provides the absolute file system path to the current workspace. e.g. for the d1-cn-os-core-deb-unstable job, $WORKSPACE = /home/hudson/work/jobs/dataone-cn-os-core-deb-unstable/workspace

#!/bin/bash
set -e

#Change this to the name of the Debian package being built
PACKAGE="SET-PACKAGE-NAME-HERE";

#Binary destination within the package. Usually the same as $PACKAGE, but not always
## IMPORTANT: check that the copy destination is correct - some of the
## debian packages don't use the convention of /usr/share/$PACKAGE
PACKAGE_DEST="$PACKAGE";

# Copy jars etc from recent builds. This is a list of all the jars / wars
# that are built by hudson and package in the .deb
# Edit this list to include all the jars and wars that need to be included.
# Path is relative to the Hudson jobs folder, which is $WORKSPACE/../..
JARS=( "PATH-TO-JAR-1" "PATH-TO-JAR-2" );

#Location where the .debs and packages will be published
#UBUNTU_UNSTABLE is a global environment variable set in the Hudson config
APTREPOS="$UBUNTU_UNSTABLE";

### Should not need to change anything past here ###

echo "Build version: $BUILDVERSION";

for CWS in "${JARS[@]}"; do
  echo $CWS
  cp "$WORKSPACE/../../$CWS" "$WORKSPACE/$PACKAGE/usr/share/$PACKAGE_DEST";
done;

#clean old builds
BUILDDIR=$WORKSPACE/build;
rm -rf $BUILDDIR;

#copy over sources for build
DPKG_DEB=/usr/bin/dpkg-deb;
mkdir -p $BUILDDIR/sources;
cp -r $PACKAGE $BUILDDIR/sources/$PACKAGE;
find $BUILDDIR/sources/$PACKAGE -name .svn | xargs rm -rf;

#Update the version info in this package
sed -i 's/^Version: [.0-9]*/&R'$BUILDVERSION'/' $BUILDDIR/sources/$PACKAGE/DEBIAN/control

#build the .deb
cd $BUILDDIR/sources && $DPKG_DEB -b $PACKAGE ..;

#remove existing versions of the .deb
find $APTREPOS -name "$PACKAGE*.deb" | xargs rm -rf;

#Copy in the new stuff
cp -f $BUILDDIR/*.deb $APTREPOS;

#Build the Packages.gz doc
APTPKG=$APTREPOS/dists/lucid/universe/binary-amd64;
mkdir -p $APTPKG;
cd $APTREPOS && dpkg-scanpackages . /dev/null | gzip -9c > $APTPKG/Packages.gz;