THDL Tools Build Systems

David Chandler has put together a build system for Jskad, the translation tool, Tibbibl, and for the fonts.  In the CVS repository, you'll find that the build system once worked for Savant and QuillDriver as well.  This document describes them, and should be updated if we put together better build systems for the two diacritics packages and for Wylie Word (but see e-mail to thdl-devel@lists.sourceforge.net that tells why we probably will continue to require WylieWord and Diacritics developers to edit from within MS Word on account of the difficulty of using CVS with VBA).

What is a build system, you ask?  A build system is a set of clear, repeatable procedures that software developers go by to compile, run, test, create documentation, and cut releases.  Our build systems are fully automated, which provides a good basis for collaborative development.  Even for individual development, a good build system gives you the confidence to make big organizational changes and saves you time in the long run.

Table of Contents

• February 2005 Update
• Glossary
• Why we chose Apache Ant
• Installing Apache Ant
• Setting things up to use CVS
• Building the Fonts module
• Building the www module -- how to publish on this very web site, that is
• Building the Diacritics and WylieWord modules
• Building the Jskad module
  • Setting things up for your first build of the Jskad module
  • Day-to-day development activities with the Jskad module
    • Testing
  • Why we chose Java Web Start and how it works
  • Cutting a Java Web Start release

February 2005 Update

As of February 20, 2005, this document is probably still pretty much up to date, but a couple of things need to be said or emphasized and that will be done in this section. David Chandler added this section quickly rather than integrating it seamlessly into the document because he's pressed for time.

The main thing to know is that we have nightly builds for the 'Jskad' CVS module (which contains more than just Jskad) running on UVA servers. (We have nightly builds for fieldling.sf.net too.) If you break those builds, then our users won't be able to try the latest versions of the software.

Another thing to note is that you probably need to tell Ant to use more memory than it does by default. If you ever get a java.lang.OutOfMemory, then try setting the environment variable ANT_OPTS to include -Xmx256m (or even 512).

Glossary

This document uses the following terms:

• CVS -- "concurrent versions system", a piece of software used by SourceForge and many others for source code revision control.
• WinCVS -- a front end for use on Windows that you can use rather than the command-line cvs tools.
• sandbox -- a directory on your local machine that mirrors one in our project's CVS repository.  You get one by using the command 'cvs checkout'.  It's called a sandbox because you can safely play in it; your work will not affect others until you commit your changes to the repository.  (Though if you really wanted to, you could create a CVS branch and then even your commits (to that branch) wouldn't affect the work of others on the trunk, or main branch.)
• ssh, scp, sftp -- the software that SourceForge uses to allow you to log in to their server without worrying about the security of your password of the subsequent communications.  ssh logs you in, scp and sftp allow you to copy files to and from thdltools.sourceforge.net.  Windows users should either get these tools with Cygwin or should install PuTTY; the latter is probably easier.
• Ant -- a.k.a. Apache Ant.  Software we use to compile and package Java programs for distrution.

Why we chose Apache Ant

Ant is built from the ground up as a full-featured, extensible build system for Java programs.  If you use Ant, your classpath-related headaches disappear (after clearing initial hurdles), and you can be sure that the build will work the same on Linux and the Mac as it does on Windows XP, because Ant itself is a Java program.  I compile, test, run, and build releases using Ant, and really like it.

Although Ant is mainly intended for building Java programs, it has enough features to make it a viable choice of build system for things non-Java, things like the Tibetan Machine Web font.

Installing Apache Ant

To install Apache's Jakarta Ant (or "Apache Ant" or simply "Ant"), you'll first need a JDK installed.  I recommend getting Sun's latest 1.4 JDK (yes, it's available for Linux too).  Next, see the Apache Ant website.  Their instructions for installation are pretty good, but you may have trouble if you're on Windows and you don't know how to set environment variables (try Control Panel->System->Advanced->Environment Variables) and the like.  One thing they don't tell you is that their binary distribution ships with a ton of documentation in $ANT_HOME/docs/ that can all safely be deleted if you care about disk space.

I recommend installing Ant yourself, and not using a package manager such as apt-get or rpm.  This is because the ordinary installation has a well-known $ANT_HOME/lib directory in which we'll wish to place a couple of JARs ( junit.jar and Vamp), as described below.  I've had personal experience with an RPM that made it impossible for me to make Ant aware of extra JARs, so stay away from RPMs and other packages.

To test your installation, use cvs to checkout either the Fonts module or the Jskad module.  The Fonts module has a simpler Ant build file (Fonts/build.xml) than the Jskad module has, so perhaps it's better for testing.  The Jskad module should be good to go as well, though, because we have a multi-buildfile system that delegates less common build activities (ones that require tweaking of your Ant installation) to supporting buildfiles and keeps the main buildfile, Jskad/build.xml, free of nonstandard constructs.  Change directory to whichever module you choose and run the following command:

ant clean

The output shouldn't alarm you, and it should conclude with "BUILD SUCCESSFUL".  If you have any trouble, see Ant's FAQs and then e-mail David Chandler.

Now that you have a good installation of Ant itself, you probably will want to install both JUnit and Vamp.  These steps can wait until later if you're not sure you'll be testing our software or packaging our software for release.  Here are the steps to install them:

• To install JUnit, first do a cvs checkout of the Jskad module (see below for info on how to use CVS).  Now, to make Ant aware of JUnit, simply copy Jskad/extensions/to-be-installed-with-ant/junit.jar to the $ANT_HOME/lib/ directory (and, if you're on UNIX/Linux, make the file world-readable).
• It isn't much trouble to install the Vamp Ant task provided by the Venus Application Publisher.  You need this to cut Java Web Start releases of Jskad, [Tibbibl if I ever get my act together], and the translation tool.  You do not need it otherwise.

  Installing Vamp's Ant task is simple enough, and is in fact simpler than Vamp's web pages of October 1, 2002 imply.  That is, if you have installed Ant 1.5.1 or later, then you can simply drop vamp.jar into Ant's lib directory (i.e., $ANT_HOME/lib/) and forget about changing your classpath (but do be sure, Linux/UNIX users, that the file is world-readable).

Your build environment is good to go!

Setting things up to use CVS

If you're working on Windows, it's probably easiest to follow SourceForge's instructions on using WinCVS and PuTTY.  The instructions there aren't up to date for the latest version of WinCVS (1.3b8 as of this writing).  Instead of telling WinCVS about plink.exe in the non-existent Ports tab of the Admin->Preferences menu, you must click the Settings button next to the server type ("ssh") and click the 'If ssh is not in the path' checkbox, and enter plink.exe's full path there.

Also note that this page is very helpful; it's about how to run WinCVS.

If you're using WinCVS 1.3b8, note that it has a bug -- it truncates the output of 'cvs diff' (as explained in this bug report).  For that reason, you may wish to use what WinCVS calls an "external diff" program, which is easy to install and configure.  Try using ExamDiff.

If you're on Linux or Unix, SF.net's site docs are excellent.  If you're on a Mac, and SF.net's docs don't do the trick, you're in trouble.

Building the Fonts module

The Fonts CVS module comes with an Apache Ant build system.  We made this choice because the Jskad CVS module already has an Apache Ant build system, and because Ant is more than powerful enough for our needs.  For this module, our only task is to cut releases.  There is nothing to compile here; the only task that a developer will face is updating the associated documentation and then cutting a new release, ready to go up on SourceForge's File Release System (FRS).

To cut releases for one or both of the Tibetan Machine Web and Tibetan Machine, do the following:

1. use cvs to checkout the Fonts module.  If you have already checked it out, run 'cvs -f update -d' (in WinCVS, hold down the shift key while selecting "Update" and then be sure the "Create missing directories that exist in the repository" box is checked).
2. make whatever changes you need to
3. be certain the the release notes (a text or html file, I believe) are under CVS control.  Perhaps you'll reuse an existing document, or perhaps you'll have to add a file to the repository.  Go to our project's download page and see how we've done release notes in the past.
4. commit those changes to the repository
5. install Apache Ant
6. change directory to Fonts and run the following command:

   ant

   If you have any problems, be sure you're using the correct version of Ant and then read Ant's FAQs and documentation.  Or you can e-mail David Chandler.  If it turns out that the problem is in the Ant build file Fonts/build.xml, make the required changes and be sure to commit them to the repository.  Now run the following command to rebuild the releases:

   ant clean dist

7. after being sure that your Fonts directory is a mirror of the one in the repository, use cvs to tag this revision that you wish to release with a name that another developer, years down the road, will recognize as referring to this release.  To do this, you can run 'cvs -f rtag tag-name Fonts' or (and this is preferred) you can run 'cvs -f tag -c tag-name'.  Good tag names are like the following:
   • Fonts_1_0 (referring to the 1.0 release of the Fonts module)
   • September_23_2002_release (referring to the release of the Fonts module made on September 23, 2002
   Note that tagging for a release of just the Tibetan Machine Web font and not the Tibetan Machine font is no different than tagging for a simulaneous release of both.
8. Now that you've tagged this release, it will be possible for any other developer to recreate the same release.  This is the heart of a good build system -- clear, repeatable procedures that aren't painful to follow.

Building the www module

Our SourceForge.net website is entirely under CVS control.  This means that we can freely make changes, knowing that they are never lost.  It also means that downloading our nightly CVS repository tarball is all that is necessary to save a backup copy of our SF.net web site.

To change an existing web page or to add a new one, you must do the following:

• First use cvs to checkout the www module.  If you've already done that, then please use cvs to update (using 'cvs update -d' if you want to create missing directories that exist in the repository) now.
• To add a file or directory, use 'cvs add'.  Change any files you wish.  Now validate your HTML using W3C's validator.  Once it passes with no errors, commit your changes.
• Now you must wait four hours for the cron job (added via crontab) to update the CVS sandbox at thdltools.sourceforge.net:/home/groups/t/th/thdltools/htdocs.  After waiting four hours, surf to the website and see for yourself.  (Note that we use pserver to update the site, so until September 2003, it may be 28 hours until you see your change.)

Updating the parts of the website not in the www module

Not all of the material up at our SF.net website is in the www CVS module, but most of it is.  Currently, the only exception is the Javadoc API documentation for Jskad, Tibbibl, and the translation tool.

Here is how to update these API docs:

• The Javadocs that THDLTools_Developers.html links to, e.g. http://thdltools.sourceforge.net/pubapi, live at /home/groups/t/th/thdltools/htdocs/.  To update them, first commit any changes you've made to the repository.  (This, combined with the date stamp that Ant puts into our Javadoc docs, will give other developers enough information to recover the revision of the source files that you used.  But feel free to 'cvs tag' the revision explicitly if you think it's important.  I tagged one revision javadocs_october_27_2002, for example.)
• Now simply run 'ant private-javadocs-dist public-javadocs-dist' from the Jskad directory, which creates Jskad/dist/docs/private-javadocs-DSTAMP.zip and Jskad/dist/docs/public-javadocs-DSTAMP.zip.  Examine the output for warnings; resolve them if necessary.  Note that the public-javadocs-dist causes warnings about links not found that should be ignored; they aren't found because the things linked to are private or package private.
• Now scp those two filse over with the command

  scp dist/docs/private-javadocs-WHATEVER.zip dist/docs/public-javadocs-WHATEVER.zip yourUserId@thdltools.sourceforge.net/home/groups/t/th/thdltools/htdocs/

  or use PuTTY's scp to do something equivalent.
• Now, shell in to thdltools.sourceforge.net with 'ssh -l yourUserId thdltools.sourceforge.net' or with PuTTY.
• Change directory (cd) to /home/groups/t/th/thdltools/htdocs/.
• Anyway, now run 'rm -fr pubapi privateapi public-javadocs* private-javadocs*' (from the htdocs/ directory).
• Now run 'mkdir pubapi privateapi'.
• Now run 'cd privateapi'.
• Now run 'unzip ../private-javadocs-WHATEVER.zip'.
• Run 'cd ../pubapi'.
• Run 'unzip ../public-javadocs-WHATEVER.zip'.
• Update the symbolic links by executing, from the htdocs/ directory, 'rm -f public-javadocs.zip private-javadocs.zip; ln -s public-javadocs-WHATEVER.zip public-javadocs.zip; ln -s private-javadocs-WHATEVER.zip private-javadocs.zip'.
• See if http://thdltools.sourceforge.net/api/ looks up to date (see the date at the very bottom of the page, for starters).  Also, verify that links to J2SE Javadoc documentation are functioning -- click on a link to java.lang.Object and be sure that it works.  If everything is fine, delete ../../private-javadocs-WHATEVER.zip and log out.
• Visit the developers' site and ensure that the four links are all functional.
• UPDATE THESE INSTRUCTIONS if something above is wrong or unclear.

That's all!

Building the Diacritics and WylieWord modules

You'll have to use MS Word to get at the Visual Basic source code in the .dot file.  Edit it there, and then commit the revised .dot file.  'cvs diff' won't work, but that's life unless you want to do a lot of cutting and pasting into the .dot file every time you make an edit.

Building the Jskad module

The following sections describe how to use the Ant build system we have for the Jskad module.

The first thing to note is that the Jskad module is misnamed.  We have all our Java code inside this module, including Jskad, Tibbibl, and the translation tool.

Setting things up for your first build of the Jskad module

You'll have to first get your computer set up to use CVS.  See above.

The next thing you'll need to do is get a CVS sandbox -- a Jskad directory on your local computer.  Use 'cvs checkout Jskad' or WinCVS's "Checkout module" command to do so.

You'll also need to checkout the Fonts module underneath the directory where you checked out the Jskad directory.  We put the TMW font into some JAR files, so the 'Jskad/Fonts/TibetanMachineWeb/' directory must exist.

Now go get the JARs you need.  Be sure to note which versions you get, because you'll need to mention that in the release notes of any releases you cut.  The following are needed:

• We use Xalan for XSLT, and its binary distribution contains JARs for Xalan and Xerces -- handy.  You only need to download Xerces' JARs separately if you need a bug fix from a later release than that included in Xalan's.  After downloading, move xalan.jar, xercesImpl.jar, and xml-apis.jar to $ANT_HOME/lib.  They need to be there to avoid loader errors when running the check-report target, which is a part of the nightly builds that also requires XSLT (i.e., the presence of those JARs, but in a place that Ant recognizes).  A consequence of using this location is that you'll probably find using targets like jskad-run much easier than specifying your own CLASSPATH.
• We use JUnit for unit testing.  See Installing Apache Ant for full installation instructions.
• We also use jdom.jar from JDOM.org.

If you need to add something else to the classpath, be it a .jar file, a .zip file, or a directory, you have several options.  You can set your CLASSPATH environment variable, or you can call Ant using 'ant -Dadditional.class.path=c:\my.jar', or you can drop JAR or ZIP files into Jskad/extensions/drop-ins.

While you're hacking, use our API docs on the web, or build your own.

Day-to-day development activities with the Jskad module

This section describes how our build system interacts with your daily work hacking on Jskad, Tibbibl, and the translation tool.

Compiling

Compiling any of the tools is a simple matter.  First, ensure that you are in the Jskad/ directory of your sandbox.

Now run 'ant program-compile' to create a directory tree Jskad/bin/program/ that contains all the .class files and needed resources (.ini files, .xsl files, et cetera).

Or you may run 'ant program-dist' to create a JAR file Jskad/dist/lib/program-vanilla.jar that contains all the .class files and needed resources, but does not contain any third-party JARs.

You may also run 'ant program-jws' to create a Java Web Start release.  See below to learn more about this.

Gotcha: We currently have the build system set up so that it builds a program, say Jskad, given merely the name of the main class, say org.thdl.tib.input.Jskad.  This means that our releases include only classes that we're actually using.  But it also means that your changes to any but the main class will not cause recompilation.  You must use 'ant clean' before recompiling.  In addition, you should clean after any cvs update.

The alternative to specifying just the main class is specifying every needed class.  If you do this, Ant often rebuilds the correct classes.  But you may have to modify build.xml when you add a new file, and your releases may include vestigial classes.

Running

Running the tools is easy for a developer.  Just run 'ant program-run', where program is one of jskad, tt, or tibbibl.  This deals with classpaths and all of that, and it compiles beforehand if needed.  Note the gotcha above if you're rerunning after changing the source code -- you'll have to use 'ant clean program-run'.  If you modify a resource, such as Jskad/source/options.txt, there are no gotchas -- Ant is smart enough to handle this.

Generating API Documentation

While you're hacking, use our API docs on the web, or build your own Javadoc API docs using 'ant public-javadocs' (to see only protected and public members) or 'ant private-javadocs' (to see all members).  They'll be waiting for you in Jskad/docs/{public,private}-javadocs/.  If you want to update the API docs on our website, follow these instructions.

Integration with IDEs such as Eclipse

Nowadays, any good Java IDE is aware of Ant and will let you use Ant for the things it is good at while using the IDE for the things it is good at.  Ask David Chandler how he got Eclipse up and running; he did it without too much trouble.

Testing

Testing is an underappreciated part of software development.  Software without a good test suite costs more money and time to maintain, and it can eventually become downright demoralizing when you can't seem to make a simple change without breaking part of the code.  If you see the break immediately, it's no big deal.  When you have a solid test suite, you're much more likely to make changes with confidence.

But testing isn't just for us, the developers.  It's mainly to give the user confidence (though hearing from us about our extensive, automated test suite is the direct method whereby the user gains this confidence).  For key parts of our system, parts that we hope others will trust with their most precious data, we need excellent testing, design, and documentation.  For other parts of the system, where failure is obvious (like GUI interfaces), testing can slide more easily.

The core modules dealing with Tibetan text are thus going to be heavily tested, mainly by unit tests and end-to-end regression tests that ensure that the software still does what we verified by hand it should do.  If we don't do this, can you imagine someone entrusting us to convert their Kangyur, in its Roman transliteration, into Unicode?  Don't imagine they will just glance at the first and last page of the converted text and sign off on our fine tool.

In short, the test suite determines whether our software is well-receieved or merely well-intentioned.

We use JUnit to provide automated unit tests.  Running them is as simple as making Ant aware of JUnit (see below) and then running 'ant clean check'.  If there are any failures, you should examine the files Jskad/TEST-* and track them down.

To make Ant aware of JUnit, simply copy Jskad/extensions/to-be-installed-with-ant/junit.jar to the $ANT_HOME/lib/ directory (and, if you're on UNIX/Linux, make the file world-readable).  More details are provide above.

To make future testing easy, you should use interfaces rather than hard-wiring in calls to classes that are difficult to instantiate.  For example, if we were to make use of a database class in our DuffPane class, doing so directly would mean that the database software would have to be properly installed and up and running in order to test DuffPane individually.  If you create an interface and use the database only through that interface, then creating a mock object that pretends to be that database but returns canned answers is a simple matter.

Modifying Jskad/build.xml

Should you need to modify the Ant build file Jskad/build.xml, feel free to ask David Chandler to do it for you.  Or just look at the contents of that file and find something similar to what you're doing.  There are a few comments in the file, and all commonly used functionality is factored out into targets that are then summoned via antcall tasks.

Why we chose Java Web Start and how it works

Java Web Start makes it possible (in theory, anyway) to install a complex application like Jskad with one mouse click of a link in your favorite web browser and one more click of a "Yes, install Jskad" button in a dialog box (necessary because our applications require security privileges to read and write from disk etc.).  Upon the second run of the program, Java Web Start asks the user if they want to create shortcuts on their computer.  Also, any time the program is invoked (from a shortcut or a web link) and a network connection is found, new versions of the software will be downloaded automatically, keeping the user up to date.

In addition, the Venus Application Publisher provides a mechanism for creating CD-ROM-based installers that load our Java Web Start code into the user's Web Start cache directory.  Then running the software will check for updates as described above.  (I don't know if we'll ever use this, and it's kind of ugly, but it's here.)

For the above scenario to play out, the user must have installed a recent (1.2 or later) Java runtime environment.  That's it from the user's point of view.

The web server that delivers the content must associate the MIME type application/x-java-jnlp-file with the "jnlp" extension and must associate the MIME type application/x-java-archive-diff with the "jardiff" extension.  In addition, if you want to be able to serve up a new release with the least difficulty (simply dropping a .war file (a Java Web ARchive) into the appropriate directory on the server), the server must run a J2EE application server such as Apache's Tomcat.

Cutting a Java Web Start release

For the reasons stated above, Java Web Start (JWS) is our distribution mechanism of choice.  But cutting a Java Web Start release is not the easiest thing you've ever done.  For one thing, because our applications require access to the local disk, they must ask for security privileges.  JWS warns users not to grant such privileges unless they know the software provider is trustworthy.  Trust depends in turn on identity, and identity is proven by public key cryptography.  So to cut a JWS release, you have to sign your JARs and rejar and sign any third-party JARs.  (If signing JARs is new to you, the J2SE documentation on the subject is okay, but hopefully reading below will tell you all you need to know.)

First, be sure you've installed Vamp's Ant task as described above.  You can do many day-to-day development activities without performing this step, so don't assume that you must have already done it.

Second, create the release notes and the changelog.  These should be two different text files.  Somewhere in one or both, you should mention the versions of the third-party JARs and APIs that we have packaged with our release.  Most importantly, put in a link to the web server on which you're going to put the web start release!  Commit these text files to the repository (adding them first if necessary).

Third, commit any changes you've made to the repository.  After being sure that your Jskad directory is a mirror of the one in the repository, use cvs to tag this revision that you wish to release with a name that another developer, years down the road, will recognize as referring to this release.  To do this, you can run 'cvs -f rtag tag-name Jskad' or (and this is preferred) you can run 'cvs -f tag -c tag-name'.  Good tag names are like the following:

• thdl_tools_1_0 (referring to the 1.0 release of the Jskad module)
• September_23_2002_release (referring to the release of all programs in Fonts module made on September 23, 2002

As you may have inferred, we release all four programs when we release any one of them.  This is only a half-truth, though--we release the unified source distribution any time we release a binary distribution, but we don't necessarily put out all four binary distributions in unison.

Now run

ant -Dkeystore="file:///c:/thdl/sandboxes/Jskad/your.keystore"
    -Dkeystore.password=baz
    -Ddgkey.password=foo
    -Dkey.alias="Johnny Developer"
    -Djnlp.codebase="http://your.web.server/THDL_STUFF"
    program-jws'

where program is one of jskad or tt.  This cleans first, so your changes will be respected.  This then creates a WAR file Jskad/dist/java-web-start/program-JWS.war.  This also extracts the signed JAR from inside that WAR, which is handy if you're deploying to a web server that isn't running a J2EE application server.  More on that below.

The argument to -Dkeystore is a URL like file:///c:/foo/keystore.  This should be a keystore that you've created with keytool.  We haven't yet paid the money to get a real X.509 key, so create your own.  Users will see a nasty warning saying that we may not be who we say we are, but so be it.  Each key in your key store has an associated alias, such as your name.  Specify it as the argument to -Dkey.alias.  The other two arguments are the password for the keystore as a whole (-Dkeystore.password) and the password for the specific key whose alias you're giving.

Inspect the output for warnings.  Ignore the warning about "[vampwar] org.apache.velocity.runtime.exception.ReferenceException: reference : template = html/test.html [line 37,column 14] : $jnlp-url is not a valid reference." but fix any others.

If you wish to deploy to a J2EE application server, such as Apache's Tomcat, now copy the .war file to the server.  You're done with the binary part of the release.

If you're deploying to a plain-vanilla web server, first ensure that its MIME type associations are correct (see above).  Now ignore the .war file and copy the appropriate .jnlp file(s) and the appropriate JAR(s) from Jskad/dist/java-web-start/ to the webserver.  Those two files are all you need, and you don't need to edit the .jnlp file because you specified the jnlp.codebase property.

Now put up a source release on SourceForge's File Release System (FRS).  After ensuring that you've made NOT EVEN ONE CHANGE to your sandbox, and after updating it, and after verifying that this revision is cvs-tagged appropriately, use 'ant src-dist' to create Jskad/source/THDL-Tools-src-todays-date.zip.  Now put this zip file up on the FRS with the appropriate release notes and changelog (which should have been committed and tagged in the repository).

You're done!  Except, of course, for the gotcha: At present, only the translation tool and Jskad have working Java Web Start releases; Tibbibl's is on the way.  Tibbibl, like Jskad, requires all-permissions security.  That means that xalan.jar, xercesImpl.jar, and xml-apis.jar [or whichever subset is used...] need to be jar-signed by the same certificate as Tibbibl.  We do this automatically via 'ant self-contained-dist' or 'ant web-start-releases', but here's how to do this manually:

1. unjar these jars.
2. delete their META-INF directory, which i think contains signature info.
3. rejar, then sign with appropriate certificate.

Now send an e-mail to thdltools-announce@lists.sourceforge.net telling users where the JWS binary release is, where the source code distribution is, and why they should upgrade or try it out.

Please e-mail us your comments about this page.

The THDL Tools project is generously hosted by: