Building Packages with Hudson
=============================


General
-------

Hudson is a continuous build environment used by DataONE and is located at:  

  http://dev-testing.dataone.org:8080/hudson



Debian Packages
---------------

Two Debian Apt_ repositories are managed by Hudson: *ubuntu-unstable* and
*ubuntu-stable*. The ubuntu-snapshot repository holds Debian packages built from
the development trunk.

The *ubuntu-unstable* repository is located at::

  http://dev-testing.dataone.org/ubuntu-unstable

and can be configured in ``/etc/apt/sources.list`` by adding an entry::

  deb http://dev-testing.dataone.org/ubuntu-unstable lucid universe

The *ubuntu-stable* repository is located at::

  http://dev-testing.dataone.org/ubuntu-stable

and can be configured in ``/etc/apt/sources.list`` by adding an entry::

  deb http://dev-testing.dataone.org/ubuntu-stable lucid universe


.. _Apt: http://en.wikipedia.org/wiki/Advanced_Packaging_Tool


Building Unstable / Snapshot Packages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The unstable or snapshot Debian packages contain code compiled from the
development trunk. These packages are rebuilt in response to commits to any
source in the hierarchy on which they are dependent, which includes the source
of the packages themselves or any of the source contributing to binaries they
contain. The general sequence for building these packages is:

1. Checkout latest copy from Subversion trunk

2. Copy the compiled binaries (usually .jar and .war files) into the
   ``PACKAGE/usr/share/PACKAGE_DIST`` folder. 

3. Copy everything into a build location, removing any SVN chaff

4. Update the Version information for the package

5. Build the .deb file(s)

6. Copy the .deb(s) to the repository location

7. Update the Packages.gz file for the repository

8. Send out notification to IRC channel #dataone-build

Steps 1 and 8 are managed by the Hudson job, steps 2-7 are performed with a Bash
script executed by Hudson, described below.

A generic script that can be modified as necessary for building a Debian package
is provided below. Key variables used in the script are listed below. Of these,
*PACKAGE* and *JARS* must be set, and *PACKAGE_DIST* should be checked for
correctness.

:PACKAGE: 

  Name of the resulting Debian package. e.g. ``d1-cn-os-core``

:PACKAGE_DIST: 

  Name of the folder under ``usr/share`` within the Debian package that will
  contain the compiled binaries. Generally the same value as *PACKAGE*, but not
  always.

:JARS: 

  An array of paths to compiled binaries that are to be added to *PACKAGE_DIST*.
  The paths are relative to the Hudson ``jobs`` folder, which is
  ``$WORKSPACE/../..``. e.g., for the ``dataone-cn-index`` job, *JARS* =
  ( "d1_cn_index_tool/workspace/target/d1_index_build_tool.jar"
  "d1_cn_index_generator/workspace/target/d1_index_task_generator_daemon.jar"
  "d1_cn_index_processor/workspace/target/d1_index_task_processor_daemon.jar" )

:APTREPOS:

  The absolute file system path to the location of the Apt repository. This
  value is set by a global environment variable (*UBUNTU_UNSTABLE*) set in
  Hudson configuration (Menu: Manage Hudson / Configure System / Global
  Properties ).

:WORKSPACE: 

  A value set by Hudson that provides the absolute file system path to the
  current workspace. e.g. for the ``d1-cn-os-core-deb-unstable`` job, $WORKSPACE
  = ``/home/hudson/work/jobs/dataone-cn-os-core-deb-unstable/workspace``

.. code-block:: bash

  #!/bin/bash
  set -e

  #Change this to the name of the Debian package being built
  PACKAGE="SET-PACKAGE-NAME-HERE";

  #Binary destination within the package. Usually the same as $PACKAGE, but not always
  ## IMPORTANT: check that the copy destination is correct - some of the 
  ## debian packages don't use the convention of /usr/share/$PACKAGE
  PACKAGE_DEST="$PACKAGE";

  # Copy jars etc from recent builds. This is a list of all the jars / wars 
  # that are built by hudson and package in the .deb
  # Edit this list to include all the jars and wars that need to be included.
  # Path is relative to the Hudson jobs folder, which is $WORKSPACE/../..
  JARS=( "PATH-TO-JAR-1" "PATH-TO-JAR-2" );

  #Location where the .debs and packages will be published
  #UBUNTU_UNSTABLE is a global environment variable set in the Hudson config
  APTREPOS="$UBUNTU_UNSTABLE";

  ### Should not need to change anything past here ###

  echo "Build version: $BUILDVERSION";

  for CWS in "${JARS[@]}"; do
    echo $CWS
    cp "$WORKSPACE/../../$CWS" "$WORKSPACE/$PACKAGE/usr/share/$PACKAGE_DEST";
  done;

  #clean old builds
  BUILDDIR=$WORKSPACE/build;
  rm -rf $BUILDDIR;

  #copy over sources for build
  DPKG_DEB=/usr/bin/dpkg-deb;
  mkdir -p $BUILDDIR/sources;
  cp -r $PACKAGE $BUILDDIR/sources/$PACKAGE;
  find $BUILDDIR/sources/$PACKAGE -name .svn | xargs rm -rf;

  #Update the version info in this package
  sed -i 's/^Version: [.0-9]*/&R'$BUILDVERSION'/' $BUILDDIR/sources/$PACKAGE/DEBIAN/control

  #build the .deb
  cd $BUILDDIR/sources && $DPKG_DEB -b $PACKAGE ..;

  #remove existing versions of the .deb
  find $APTREPOS -name "$PACKAGE*.deb" | xargs rm -rf;

  #Copy in the new stuff
  cp -f $BUILDDIR/*.deb $APTREPOS;

  #Build the Packages.gz doc
  APTPKG=$APTREPOS/dists/lucid/universe/binary-amd64;
  mkdir -p $APTPKG;
  cd $APTREPOS && dpkg-scanpackages . /dev/null | gzip -9c > $APTPKG/Packages.gz;