GODI User's Manual

Gerd Stolpmann

(Last updated: October 21, 2004)

Table of Contents

Preface

GODI is a programming environment for the computer language Objective Caml (O'Caml), a result of the research activities of the Cristal project at the INRIA institute in France.

From INRIA you can get the O'Caml compiler and runtime system, but this is usually not enough to develop applications. You also need libraries, and fortunately there are many developers all over the world providing them; you can go and pick them up. But it is a lot of work to build and install each of them.

GODI is a system that simplifies this task: It is a framework that automatically builds the O'Caml core system, and additionally installs a growing number of pre-packaged libraries. For a number of reasons GODI is a source-code based system, and there are no precompiled libraries, but it makes it very simple for everybody to compile them.

Features of GODI: This manual is a mixture of a user's and developer's manual, and you find information at various levels of details.

Chapter 1  Getting and Installing GODI

GODI is an O'Caml distribution that is compiled and installed from sources. The main reason for this is that it simplifies the maintenance of the GODI system from the perspective of the developers. GODI has only very little central infrastructure, more or less it is only a file management and distribution service, and complicated installations like compile farms are not needed. Furthermore, GODI supports a wider range of operating systems than it would be possible for a binary distribution. Users will also note the advantage that software updates happen frequently, and the distributed software versions are usually the latest that are available.

Of course, there is also a downside, especially GODI users must do more than for distributions where the software is available in binary form. First, the basic «C toolchain» to build C programs must be already installed on the system, and it must work. Second, the development parts of the system libraries must be installed on the system, e.g. C header files. The users must also recognize by themselves when additional external libraries are required for certain GODI packages. Third, the users' patience is sometimes stressed, as it takes a bit of time to build software. Fourth, the users should be prepared that sometimes things go wrong. The GODI developers cannot guarantee that the distributed build procedures work on every system. Sometimes they make too optimistic assumptions, for instance it is sometimes expected that the OS has features it actually does not have (e.g. that shell utilities have GNU extensions). Such errors happen from time to time, and the developers are glad when the users inform them about such mistakes. Of course, this would require that the users have some skills recognizing them. Summarised, the GODI users need some basic skills in system and software build management.

Resources

There are a number of on-line resources that are important for users. Links are available on the GODI homepage, http://godi.ocaml-programming.de:

1.1  Preparing your system

It is very likely that you must first install software because you can even start using GODI. This is the generic list for all OS: Of course, these are only the requirements for the minimum GODI installation. Depending on which GODI packages are installed, further external software may be needed.

There is a more detailed list in the README file contained in the GODI bootstrap tarball. It depends on the OS which of the tools come with the OS and which must be additionally installed.

1.2  Bootstrapping

The bootstrap procedure installs the minimum GODI system. Currently, the bootstrap procedure has two stages that must be performed one after the other. Stage 1 installs basic tools written in the C language that are required for the GODI package system. This means, after stage 1 the installed software is managed in the form of packages. We will discuss later what a package really is, for now just think a package as a group of files that is added to the system as a whole, and that can also be removed from the system as a whole.

Stage 2 installs the minimum O'Caml environment, and further parts of GODI.

In order to bootstrap, you need the bootstrap tarball, see http://godi.ocaml-programming.de for where it can be downloaded. Basically, you execute the following steps:
  1. Extract the bootstrap tarball:
    gzip -d godi-bootstrap-<VERSION>.tar.gz 
    tar xf godi-bootstrap-<VERSION>.tar 
    cd godi-bootstrap-<VERSION>
  2. Enter stage 1:
    ./bootstrap --prefix <PREFIX>
  3. Adjust PATH, optionally edit <PREFIX>/etc/godi.conf:
    PATH=<PREFIX>/bin:<PREFIX>/sbin:$PATH 
    export PATH
  4. Enter stage 2 (a working Internet connection is required):
    ./bootstrap_stage2
The bootstrap procedure is discussed in more detail in the README file contained in the bootstrap tarball, especially what to do when things do not work as expected.

As <PREFIX>, you can choose any empty or not yet existing directory. It is not possible that <PREFIX> points to an already used directory like /usr/local as GODI needs its own private directory hierarchy.

It is not recommended to install GODI as super-user. Either install GODI privately under a non-privileged account, or create a special «godi» account for a shared installation.

Options

The bootstrap script (stage 1) has a number of command-line options (needed for adjustments in very special situations):

1.3  Installing packages with godi_console

After the bootstrap procedure has been finished successfully, a number of programs are installed in <PREFIX>/bin and <PREFIX>/sbin. The former directory contains applications that may be executed by everybody whereas the latter directory is reserved for administration programs. One of these is godi_console which serves as the central management tool. It goes into an interactive mode when called without arguments.
+-------------------------------- GODI Console --------------------------------+ 
                                                                                 
      > > > Select Source Packages < < < 
                                                                                 
      FL NAME               INSTALLED  AVAILABLE  COMMENT 
==========Packages available as source code:==================================== 
[  1]    apps-camlmix       1.1        1.1        Processes macros written in pu 
[  2]    apps-cduce         0.2.1      0.2.1      XML-oriented functional langua 
[  3]    apps-cduce-cvs                20040829   XML-oriented functional langua 
[  4]    apps-godiva        0.9.2      0.9.2      High-level tool for simplifyin 
[  5]    apps-headache      1.03       1.03       Tool for managing headers in s 
[  6]    apps-ledit         1.11       1.11       Line editor wrapper 
[  7]    apps-schoca                   0.2.0      Scheme interpreter written by 
[  8]    apps-unison        2.10.2     2.10.2     File synchronizer 
[  9]    base-curl          7.11.2#2   7.11.2#2   The version of CURL for GODI 
[ 10]    base-expat         1.95.7#3   1.95.7#3   The version of expat for GODI 
[ 11]    base-gdbm                     1.8.3#4    The GNU database manager 
[ 12]    base-pcre          4.5#1      4.5#1      The version of PCRE for GODI 
[ 13]    base-subversion-c$            1.0.6      The subversion client allows d 
[ 14]    conf-curl          3          3          Configures which curl library 
[ 15]    conf-expat         6          6          Configures which expat library 
[ 16]    conf-freetype2     1#1        1#1        Configures which freetype2 lib 
----------------------------------------------------------------(more)---------- 
[p]rev [n]ext [u]pgrade all [s]tart/continue installation [h]elp e[x]it menu 
>

Figure 1.1: Selecting source packages with godi_console

After starting godi_console type «2» to enter the menu «Select source packages». The list of available and installed packages appears (shown in figure 1.1). You can select (and deselect) packages by entering the number and typing «b» to build the package, «k» to keep the package as it is, or «r» to remove the package (in the corresponding submenu). Finally, press «s» to start the installation, a procedure comprising the following steps: This means, godi_console guides you through the build and installation processes, without having to enter commands. We do not discuss godi_console here in detail, as it is equipped with self-explanatory help texts.

A note for advanced users: godi_console also features a command-line mode which is useful for scripting purposes. There is a manual page for godi_console explaining this mode.

Warning: Avoid to install new packages when disk space is already tight. You may run into situations where it is hard to recover from. Furthermore, it is a bad idea to stop godi_console while a package installation is being performed.

1.4  Configuring external libraries

Unfortunately, godi_console cannot build everything in an automatic way. Especially one point requires manual intervention from time to time: External C libraries.

Of course, it is required that these external libraries are installed before GODI can make use of them. For example, if you install the package godi-zlib it is a good idea to check whether the underlying C library libz is already present or not (don't forget to check that the C header files are also installed). Especially for non-free OS like Solaris there is no standard place to install such external libraries. Some admins put them into /usr/local, some into /opt, and a lot of further private locations are in use, too. For Linux and BSD, however, these libraries are often part of the OS, and can be found at known places in /usr.

The good news is that GODI is very flexible regarding these locations. There are two ways of telling GODI where to find libraries: By changing the global configuration, and by setting package-specific parameters.

The global parameter in question is SEARCH_LIBS: For example, when libz is installed in /opt/phantasy such that the library libz.so is located in /opt/phantasy/lib and the C headers are in /opt/phantasy/include, one can tell GODI this place by setting
SEARCH_LIBS += /opt/phantasy
in the global configuration file <PREFIX>/etc/godi.conf. This parameter is respected by most packages that need external libraries (but not by all, as there are other, incompatible methods of looking up libraries, see below).

GODI already knows a number of standard locations for libraries, i.e. directories where certain OS install libraries by default. For example, NetBSD usually installs add-on libraries in /usr/pkg which is already part of the built-in knowledge.

You may have already noticed that there are conf-<NAME> packages for a number of external libraries <NAME>. For example, there is a conf-zlib package. The role of these packages is to find and store the configuration where external libraries are expected to be found by GODI. By default, the conf packages iterate over the directories enumerated by SEARCH_LIBS, and look for the needed libararies in these places. Furthermore, a small test program is tried to build, just to see whether the found libraries really work.

Sometimes the library cannot be found, or additional compiler or linker flags must be set. The conf packages allow you to set the individual configuration parameters. Of course, the GODI user must already know which parameter must be set to which value -- in other words, this is tweaking for experts. In godi_console, one can set the configuration parameters by going to the configuration screens of the conf packages. For example, conf-zlib has two such parameters:
GODI_ZLIB_INCDIR:
The directory where the C header file zlib.h can be found
GODI_ZLIB_LIBDIR:
The directory where the library file libz.so can be found
Some conf packages also allow you to set the flags for compiling and linking directly rather than setting directories. In any case, the individual configuration parameters override the search strategy followed by default.

Nowadays, libraries are more and more shipped with special configuration scripts. These scripts simply output the required compiler and linker flags. When available, GODI prefers these scripts, and sees them as a trusted source whose knowledge can be expected to be right. For example, the freetype library has such a script, freetype-config, and one can call it by «freetype-config --cflags» to get the compiler flags, and by «freetype-config --libs» to get the linker flags. The corresponding GODI configuration package is conf-freetype2. Instead of searching the library directly, it just looks for where this configuration script is installed. Normally, the locations in SEARCH_LIBS are checked (by looking into the bin subdirectories), and the directories in PATH are checked. If the script cannot be found, it is still possible to set the location directly with a configuration parameter:
GODI_FREETYPE2_CONFIG:
The absolute path to the freetype-config script
When such a script is used, the configuration package does not support to specify the directories of the library directly, or to set the flags.

As pointed out, the output of the configuration scripts is simply trusted. If it happens that the obtained flags do not work, the script is wrong, and it is not possible to use the library from GODI.

Many OS now use the ELF file format for libraries (e.g. Linux, BSD, Solaris). ELF allows several ways of finding libraries at runtime, i.e. when the program using the library is started: For a number of reasons, GODI never uses the LD_LIBRARY_PATH feature. This variable is more an ad-hoc solution to get misconfigured libraries working, but not the appropriate means for a permanent and professional environment like GODI.

GODI uses one of the other two options: First, it is checked whether the library can be found by keeping the default settings, and only if this does not work, the RPATH feature is enabled. Note that this automatism is not applied when a configuration script is used; in this case it is expected that this script already knows which way is the right one to find the library at runtime.

Important note: Currently, this approach is not put through at all places. This has the effect that there are usually more RPATH settings than needed. This is rarely problematic, but I already had the case that a system library was installed in two versions, and the RPATH setting was wrong. In particular, this library was libGL.so, and one version (in /usr/lib) was the MESA software 3D rendering version, and the other version (in /usr/X11R6/lib) was the hardware 3D rendering version. Because of implementation errors, stub libraries were installed with RPATHs for /usr/lib (which is totally useless), and my programs suddenly loaded the GL library for software rendering. The workaround in such cases is to byte-compile with -custom (or to use the ocamlopt compiler), and to fix the RPATH with -cclib -Wl,-R/preferred/path. This GODI problem will not be fixed soon.

1.5  Using libraries from base packages

Usually, GODI does not include add-on C libraries like libz; it is expected that these are already available by the OS, or that the sysadmin has already installed them. Sometimes, however, GODI needs certain versions of the libraries, and it would be painful to require that the sysadmin updates the OS or other parts of the system only to make GODI happy. In these cases, GODI includes the libraries to install in the «base» series of packages, e.g. base-pcre includes the preferred version of the PCRE library.

These packages are not enabled by default, however. It is first checked whether the library version provided by the OS or the version found somewhere on the system is acceptable. If not, the conf package fails, but prints a hint that enabling the base package would be a simple solution for the problem. By setting a configuration parameter, the GODI user can do this. For example, the conf-pcre package can be made using the base-pcre package by setting
GODI_BASEPKG_PCRE=yes
The rest is again fully automatic.

Chapter 2  Using GODI

In this chapter, I would like to give some hints for O'Caml beginners, and explain where to find what in the GODI environment.

2.1  Starting the O'Caml toploop

The O'Caml bytecode compiler can be called as a so-called toploop: The user can enter declarations and expressions, and these are immediately compiled to bytecode, and immediately executed. The toploop is very handy for coding attempts, and also an interesting debugging aid for larger programs (because one can load already compiled bytecode into the toploop, too). One can invoke the toploop with the command ocaml:
$ ocaml 
        Objective Caml version 3.08.1 
#
For example, define the faculty function as (note that the # at the beginngin of the line is the prompt symbol, and that the ;; at the end of the line indicates the end of the user input):
# let rec fac n = if n <= 1 then 1 else n * fac(n-1);;
The toploop answers with:
val fac : int -> int = <fun>
This means that fac is a function taking integers as input, and returning integers as results. Call the function as
# fac 10;;
and you get the result 3628800. Of course, we cannot give here an introduction into the O'Caml language, so we stop here. It is recommended to install the package godi-ocaml-manual which includes both introductory and reference documentation of the O'Caml language. The installed manual can be found in the directory <PREFIX>/doc/godi-ocaml-manual.

You can exit the toploop by pressing CTRL-D or typing #quit;; (including the hash mark).

You may have noticed that the toploop does not include a line editor. To get around this limitation, install the package apps-ledit, and call the toploop by
$ ledit ocaml
This enables a number of keys (cursor keys, delete key, etc.).

2.2  A simple IDE: ocamlbrowser

The O'Caml core distribution includes a simple IDE that allows you to explore libraries, to edit O'Caml sources, and to run the toploop: ocamlbrowser. As this program uses the Tk library for the GUI operations, it is not installed by the GODI bootstrap procedure (so it is not necessary to deal with the complications of finding external libraries already at this early stage). ocamlbrowser is contained in the godi-ocaml-labltk package.

The program ocamlbrowser is invoked without argument, and pops up a new window with three columns. In the leftmost column, the modules of the standard library are listed. If you click at a module, the middle column shows the definitions of the module. The rightmost column is only used when nested modules occur (e.g. MoreLabels.Set).

If you click at a definition (v=value, t=type, cn=exception, m=module, ...) the contents of the definition are shown below the three columns. You can also view the interface and the implementation files, if available, by pressing the buttons «Intf» and «Impl», respectively.

The toploop is invoked with the menu entry «File → Shell...». It works like the ordinary toploop but also does syntax highlighting.

The editor is invoked with the menu entry «File → Editor...». It performs syntax highlighting, and you can even typecheck your definitions («Compiler → Typecheck»). As a special feature, you can query the types of subexpressions after a typecheck pass: Just put the cursor near the interesting symbol, or mark the expression, and press the right mouse key.

By selecting «Edit → To shell», the current definition (or the marked region) is copied over to the shell window (if open).

Although ocamlbrowser has some interesting features, it is still too limited in order to be useable as professional developement environment.

2.3  Using emacs/xemacs

GODI does currently not include packages with the required emacs Lisp definitions.

2.4  The O'Caml compilers and tools

The O'Caml core distribution includes: These tools are all described in the O'Caml manual. The native-code compiler is not available for all platforms, as well as the .opt versions of the tools.

The GODI version of ocamlmklib is a wrapper script around the real ocamlmklib.bin tool that adds a number of default options that should be present in a GODI environment.

In addition to these official tools, some less official tools are also installed:

2.5  Using add-on libraries with findlib/ocamlfind

Unfortunately, the O'Caml core distribution handles libraries in a rather low-level way, and the user of the libraries must know the directories where these are installed. Furthermore, the dependencies between the libraries must be manually resolved. This style is similar to the way libraries are handled in the C language.

The findlib library (and the command-line frontend ocamlfind) try to bridge the gap between the user's needs and the low-level but robust approach of the O'Caml core. GODI equips all libraries with the necessary meta information findlib needs to process the libraries, and to make them available in a user-friendlier manner.

In the toploop, findlib can be enabled by the directive (also type the hash mark):
#use "topfind";;
This installs a number of additional toploop directives: For example, a single
#require "pxp";;
loads the XML parser PXP into the toploop, including all predecessor libraries PXP is dependent on.

Unfortunately, some platforms cannot load libraries into the toploop that depend on external C libraries. For example, Cygwin and NetBSD are such platforms. When you try to load such libraries, these platforms output the error
Cannot load required shared library: 
dynamic loading not supported on this platform.
The workaround is to create a custom toploop that statically links the needed libraries. For example, to create a custom toploop with support for PXP, run the command
$ ocamlfind ocamlmktop -o mytop -package pxp,findlib -linkpkg
which creates a toploop program called mytop which can be used instead of the pre-built toploop ocaml. Note that findlib must always be mentioned as package. The toploop mytop has already built-in support for findlib, so you need not to «#use» topfind at the beginning of every session. The «#require» directive for PXP is still necessary, however.

In order to call the standalone compilers ocamlc and ocamlopt, the tool ocamlfind should be used. This is a wrapper program around the compilers that adds a number of additional command-line options. For example, to compile the module sample.ml that calls functions of PXP, use
$ ocamlfind ocamlc -c sample.ml -package pxp
When linking executables, the option -linkpkg must be passed to indicate that the libraries must be linked, too:
$ ocamlfind ocamlc -o sample sample.cmo -package pxp -linkpkg
The tool ocamlfind can also be used as wrapper for ocamlcp, ocamlopt, ocamlmktop, ocamldep, ocamldoc, and ocamlbrowser. The latter is very convenient to browse the interfaces of add-on libraries, e.g. to view the definitions of PXP, run
$ ocamlfind ocamlbrowser -package pxp-engine
(Note that we refer to pxp-engine, and not pxp, as the latter is only an empty pseudo package, and the ocamlbrowser call does not resolve dependencies.)

A special feature of findlib is that it simplifies the usage of camlp4 enormously, the grammar-level preprocessor for O'Caml. In order to enable camlp4, set the -syntax option:
$ ocamlfind ocamlc ... -syntax camlp4o
This enables camlp4 with standard syntax. Replace camlp4r for camlp4o to get the revised syntax. The interesting feature of ocamlfind is that one can easily specify camlp4 extensions. For example, to get the xstrp4 extension, just use
$ ocamlfind ocamlc ... -syntax camlp4o -package xstrp4
i.e. add such extensions simply to the list of included packages.

The findlib library is available as GODI package godi-findlib. It is installed as part of the bootstrap procedure. There is, however, a small but useful option that is not enabled by default, and requires a recompilation of findlib: The Makefile wizard. This is a GUI to create findlib-aware Makefiles with a few clicks.

To get it: Start godi_console, select the godi-findlib package, and enter the configuration menu. Set the parameter
GODI_FINDLIB_TOOLBOX = yes
and rebuild findlib. You will also need godi-ocaml-labltk (and thus tcl/tk) in order to build this special version of findlib (which is the reason why this option is disabled by default). The result is that the «Makefile wizard» is included in the findlib package. Call the wizard with
$ ocamlfind findlib/make_wizard
and follow the instructions in the new window that pops up.

There would be a lot more to say about findlib. As part of the package, the findlib manual is also installed, so please look there for more information.

A final note on the syntax «-I +pkgname» the O'Caml compilers implement themselves. It was introduced as simple mechanism to locate add-on libraries. For GODI, this kind of referring to libraries should be regarded as deprecated legacy mechanism. The point is that this syntax is much less flexible than findlib, and that it is also dictates the directory where the library must be installed (which is not acceptable).

2.6  Finding package documentation

The documentation for package P can be found in <PREFIX>/doc/P. The preferred format is HTML.

There is also a small CGI that can display library interfaces: godi-findlib-browser. It can be found in <PREFIX>/doc/cgi-bin/browser.cgi after installation. One can easily view all interfaces (but w/o formatting), and there is also a full-text search option. In order to activate this CGI, you need a properly configured web server. For Apache, the widely used web server, it is usually sufficient to include the directives
ScriptAlias /godi-bin <PREFIX>/doc/cgi-bin 
<directory <PREFIX>/doc/cgi-bin> 
  AllowOverride None 
  Options ExecCGI 
  Order allow,deny 
  Allow from all 
</directory>
into the configuration file httpd.conf, and to restart the web server. The CGI becomes visible under the URL http://servername/godi-bin/browser.cgi. There are also other ways of configuring Apache for this purpose.

2.7  Key packages

Some packages play a special role in the package system:

Chapter 3  The Architecture of GODI

In this chapter, we look at the various parts of GODI, and how they are related to each other. It should become clearer how GODI works, and what one can expect from it.

3.1  GODI servers and clients

In principle, GODI is a kind of client/server system. This aspect is usually overlooked, although it is one of the most important properties. The GODI system, as it was installed in chapter 1, consists only of the client part that controls the locally installed packages. In addition to this, there is also a GODI server that provides the necessary information which packages exist and what the packages contain. The GODI client (e.g. godi_console) may contact the server to update the list of packages. In godi_console you can trigger this by selecting the menu item «Update the list of available packages».

The GODI server is mainly a Subversion repository where the files are stored that make up the various packages. This repository is maintained by the GODI developers in a collaborative effort. You can view this repository under the URL https://gps.dynxs.de.

The godi-build directory contains the packages, whereas godi-bootstrap contains the base software including the bootstrap script (from which it derives its name). (The other directories contain software not related to GODI, although some of the libraries are available as GODI packages.)

More precisely, the godi-build directory on the GODI server only provides the build instructions, i.e. the set of rules that control the build and installation procedures. The GODI server does not store the distribution files, i.e. the tarballs made and distributed by the authors of the software packages. These are downloaded from the primary http or ftp servers. (To be even more exact, the GODI server keeps copies of the distribution files, but these are only used when the primary servers are not reachable.) For example, the package godi-ocamlnet in version 0.98 has the following build instructions and distribution files: The build instructions also contain dependency information. For this reason, it is necessary that the build instructions of all packages must be available at any time, and because of this, godi_console updates them in a single step. The distribution files, on the contrary, are only downloaded when the corresponding package is built.

3.2  Local directory layout

The GODI directory hierarchy follows Unix conventions with some additions:

3.3  Packages

We already said that a package is a group of files that is installed as a whole. This is a simplified definition, and when one looks in detail at the package concept, it becomes clear that the same package may exist in three ways: The metainformation includes:

3.4  Libraries

As already pointed out, the O'Caml libraries always support findlib in the GODI system. This is not a very hard requirement, and it is usually simple to even add such support to libraries where the author does not do this. Findlib bases only on a few concepts, and complicated situations cannot arise.

The key ideas are that libraries are stored in known directories (directory convention), and that there is a file with metainformation about libraries (called «META»). The directory convention is as follows (here shown in the way GODI realises it): Of course, it is possible that the same library is installed under both pkg-lib and site-lib. In this case, site-lib has precedence. Anyway, it is a bad idea to do so, because this may break GODI's build system. In general, it is ok to have libraries in site-lib that depend on libraries in pkg-lib, but not vice versa.

In addition to the directory convention, findlib manages libraries also by storing metainformation about the libraries. These are put into files with the name META, and the META files are contained in the code directories of the libraries. META has usually only a few lines, e.g.
description = "The library foo" 
version = "1.0" 
archive(byte) = "foo.cma" 
archive(native) = "foo.cmxa"
and is seldom more complicated.

Findlib can also express dependencies (library X depends on library Y = library X uses features of Y). This mechanism is different from the GODI package dependencies, and there is no strict need that the dependencies of findlib and GODI correspond to each other, although this is usually the case for obvious reasons.

O'Caml libraries linked with external C libraries are a special case. In principal, there is an O'Caml part, and a C part. As shown in section 1.4, it is required that the GODI user can configure where the C library is located. The configuration data are contained in the configuration package conf-foo for the external C library foo. The whole story is as follows: The O'Caml library and the stub library are part of the GODI package godi-foo. The location of the external C library is specified by conf-foo. It is now possible that the C library is located outside of GODI, or that the C library is available as GODI package, too. In the latter case, the package is called base-foo, and it is required that the C library is installed in the directory <PREFIX>/lib.

Note that although the information where the external C library resides is specified in conf-foo, these locations are also entered into the O'Caml library as part of the build process, so that they are also available in godi-foo. At runtime, conf-foo is no longer needed. Furthermore, if the user wants to change the configuration, it is not only required to build conf-foo again, but also godi-foo, and GODI does not remind you of that.

Some platforms allow that C libraries are dynamically loaded into the running bytecode interpreter, but in general this cannot be assumed. Of course, the C libraries must be available as DLLs (or DSOs in Unix terms) in order to be dynamically loadable. When the platform does not support this technique, however, there is no advantage to have C libraries in DLL form, as O'Caml links statically anyway.

Linking with C libraries is really complicated, and there are a number of details that must be handled differently for the various platforms. It is currently not clear which facts about linking are important for GODI users, and which are really technical details only experts need to know.

Chapter 4  Managing a GODI Installation

4.1  What can be done with godi_console

The following tasks can be easily carried out with the help of godi_console: It is recommended to check whether enough disk space is available before installing packages. One can get into problematic situations when the disk becomes full in the wrong moment, and it is difficult to recover from this. Furthermore, it is a bad idea to stop godi_console in the wrong moment (CTRL-C) because of the same reasons. The critical step begins when godi_console prints that it is installing a package («===> Installing for package»), and ends after the installation has been registered («===> Registering installation for package»).

4.2  Installed packages

The command godi_info may be used to get detailed information about installed packages: Note that godi_info -F does not work (looking up the package by file name), because the needed reverse index is not available.

The command godi_delete may be used to remove a package from a system (you can also do this with godi_console). Give the -r option to delete the package and all successor packages.

4.3  Binary packages

As mentioned earlier in this manual, the binary package file is put into the directory <PREFIX>/build/packages/All after the build of a package has succeeded. This means that this directory always contains a complete copy of the current installation plus a lot of history information.

When a package is updated to a new version, the binary package file of the old version is not deleted. In principle, this allows you to go back, and to restore the old package. However, the history function is limited by the fact that often package files are overwritten when a package is rebuilt for a different equipment of predecessor packages, but for the same version of the software. For example, consider that there is a package foo in version 1, and a package bar in version 1. Furthermore, bar is dependent on foo. When a new version of foo is released, e.g. version 2, and the GODI user updates the package, both packages are built again, foo in version 2, and bar in version 1. The result is that a new binary package file is created for foo, namely foo-2.tgz, but that the old package file for bar is overwritten, because it is still called bar-1.tgz.

The main purpose of the binary packages is to simplify the distribution of software in LANs. One can copy the packages from one system to another, and install them, which is a lot simpler than to build the software on every system from source.

Tasks related to binary packages:

4.4  Restoring old packages

In principle, one can restore binary packages (i.e. install the old binary package file again), and one can recover source packages (i.e. go back to an old version of the source package, and build it again).

4.4.1  Restoring an old binary package

In principle, you can restore an old binary package by calling godi_add -u for it:
cd <PREFIX>/build/packages/All 
godi_add -u <NAME>.tgz
Sometimes this does not work, however, because the old package is not compatible with the current set of predecessor or successor packages (godi_add reports a conflict). You can try to replace the problematic packages by historic versions, too, but this is a very complex task, and it is even unclear whether it is possible at all. As explained in section 4.3, package files may be overwritten in certain circumstances, and this means that the required old files might be lost.

In general, it is better to build the old version of the package from source, see the next section for instructions.

4.4.2  Building an old version of a package

This way of restoration is usually possible. In <PREFIX>/build/buildfiles, GODI stores the build instructions of all package versions that were ever downloaded. In <PREFIX>/build/distfiles, GODI stores the distribution files of all package versions that were ever built.

Follow these steps to install the package corresponding to the build instructions in <CAT>-<NAME>-<VERSION>.build.tgz:
  1. Change the directory:

    cd <PREFIX>/build/<CAT>

  2. Delete the current build instructions (if present):

    rm -rf <NAME>

  3. Extract the old build.tgz archive:

    godi_tar xzf ../buildfiles/<CAT>-<NAME>-<VERSION>.build.tgz

  4. Start godi_console and build the package again (see above). It might happen that the distribution files do not exist locally in the distfiles directory, because the package was never built before. In this case, godi_console tries to download them from the Internet. There is no guarantee, however, that historic distribution files remain available forever.

    Of course, godi_console may want to rebuild other packages, too. This is just the same mechanism as building new packages. It may happen, however, that the already installed packages are too modern for the new package. This is not detected by GODI in advance because this kind of information is not available. The most likely symptom is that the build fails with a compiler error.
In order to keep this possibility, it is strongly discouraged to delete the files in the buildfiles and distfiles directories.

4.5  Distribution upgrades

There are usually several «release lines» of GODI, i.e. several completely independent series of packages. When you perform the bootstrap procedure, one of the release lines is automatically selected and installed (there is always one release line reflecting the most up-to-date state). Up to now, the following types of release lines are in use: Currently, the existing release lines of the first type are «3.07» and «3.08». The release lines of the second type are announced in godi-list.

The release line GODI uses is determined by the variable GODI_SECTION that can be set in the file <PREFIX>/etc/godi.conf. After changing this variable, please note that GODI does not really detect that a different distribution is selected, it just retrieves the package from the newly selected release. This means: The consequence is that the change of the release line works only correctly when this implies an upgrade, i.e. the packages of the new release are newer. If you really want to change to an older release, you must first delete all extracted build instructions:
cd <PREFIX>/build 
rm -rf apps/* base/* conf/* godi/*
After that, at least the build instructions of the older release are downloaded, and you can start to build packages. You might still encounter strange effects, though, because package downgrades are a somewhat hairy operation, as necessary information about compatibility between old and new packages are missing.

Chapter 5  Packaging Software

This chapter explains the core method of packaging software. GODIVA is another method with a simpler package specification file. GODIVA processes this file, and generates build instructions according to the core method, so GODIVA should be regarded as a higher layer that works on top of the core layer.

The question is whether to use GODIVA, or to manually apply the core method. There is no simple answer, but the following criterions may help: Some of the limitations can be worked around by patching the result of the generated build instructions, however.

If you think you may want to try GODIVA, just install the apps-godiva package, and read more on the homepage: http://projects.phauna.org/godiva. The following information may be still of interest, however.

5.1  The build directory

The build instructions for a package are stored in the build directory:
<PREFIX>/build/<CAT>/<CAT>-<NAME>
You find here the following files and directories: Usually, one needs only Makefile, DESCR, and PLIST (distinfo is generated, see below).

There may be the directory work as well. It contains the unpacked software, and this is the place where the build process really happens. By deleting work, one can reset the build process to the beginning. Note that it is possible to configure a different place for work (e.g. somewhere in /tmp).

5.2  Stages

The build process is structured into several stages. A certain stage can only be reached from the immediately preceding stage (but there is some shortcut logic, see below). The stages are: Normally, the build process just «climbs» the stages one after the other. The build process remembers the already reached stages by placing invisible files into work, e.g. work/.install_done indicates that build has completed the install stage.

When developing a package, it is possible to go to a certain stage, and to check whether everything has been done right (in godi_console, the stages are simply iterated, and one does not have the chance to stop and to check what is going on). This is done by calling godi_make, and passing the name of the stage, e.g.
godi_make configure
continues the build process until at least the configure stage is reached.

There are defined actions that are carried out for every stage. These actions can be configured by setting variables in Makefile (explained below). In addition to this, one can define pre and post actions for every stage by adding rules to Makefile, e.g.
pre-configure: 
        <commands>
causes that these commands are executed before the predefined actions of the configure stage. In the same way, a post-configure rule would be executed after the predefined actions of the configure stage.

5.3  The Makefile

The minimum Makefile looks as follows:
.include "../../mk/bsd.prefs.mk"  
VERSION= ... 
PKGNAME= ...-${VERSION}  
PKGREVISION= ... 
DISTNAME=... 
DISTFILES=....tar.gz  
CATEGORIES=...  
MASTER_SITES= ... 
MAINTAINER=... 
HOMEPAGE=... 
COMMENT=...

.include "../../mk/bsd.pkg.mk"
Of course, this is only rarely enough, but often only a few additions are needed. We will discuss possible additions below. The «.include» directives are mandatory, and load the rest of the build framework. The variables have this meaning: With only these variables, the stage actions are defined as follows: We will now discuss how to define a number of frequent variations of this default. As most configurations can be done by setting further variables, the reference document of the variables is quite important: The file makevar-ref.txt describes the variables. It is usually installed in <PREFIX>/doc/GODI/project-doc.

5.3.1  Customising the «extract» stage

There are not really many ways for this type of customisation. If only some DISTFILES need to be unpacked, one can set

5.3.2  Customising the «patch» stage

Patches need not to be declared in Makefile (only in distfiles). The patches must be put into the patches directory. Usually, the file names of patches have the convention
patch-<letter><letter>-<info>
where the two letters define the order in which the patches are applied. The info suffix is just a descriptive string, e.g. one can mention the patched file (if it is only one file).

Patches are applied relative to the toplevel directory of the unpacked sources. The patch format should be «unified» patches. It is not allowed that files are created or deleted by patching; in the first case one must copy the files from the files directory to the source tree, and in the latter case one must delete the files by a script.

How to create patches

In this example we create a patch for the file foo.txt which is part of the bar-3.14 package.
  1. Copy the original version of the file:
    cd work/bar-3.14 
    cp foo.txt foo.txt.orig
  2. Modify foo.txt as required
  3. Create the diff file (current directory is still work/bar-3.14):
    diff -au foo.txt.orig foo.txt >../../patches/patch-aa-foo.txt
If the file to patch is located in a subdirectory, do not change to this subdirectory! Create the difference always from the toplevel directory of the unpacked sources.

Finally, you should also update distfiles, see below how to do this.

5.3.3  Customising the «configure» stage

As mentioned, the default is not to configure the software. The following variables enable this:

5.3.4  Customising the «build» stage

One can set a number of variables, and there are also some typical «code snippets». Often, O'Caml software must be built with «make all» to get the bytecode version, and «make opt» compiles to the native code version (if supported). This is usually expressed by this code snippet
.if ${GODI_HAVE_OCAMLOPT} == "yes"  
ALL_TARGET= all opt  
.else  
ALL_TARGET= all  
.endif
(or some variation). The variable GODI_HAVE_OCAMLOPT expands to yes when the ocamlopt compiler is available, and to no otherwise. (Note that there are some more variables that describe the properties of the O'Caml core, see the variable reference.)

When findlib is used by the software, a special configuration must be enforced. This configuration sets a different lookup path for libraries such that the GODI-managed libraries have precedence. To get it, put this line into Makefile:
MAKE_ENV+= ${BUILD_OCAMLFIND_ENV}
If forgotten, it may happen that the build fails because a required library is not found, or in the wrong version.

5.3.5  Customising the «install» stage

Many of the «build» variables are also applied in the «install» stage, namely USE_GMAKE, MAKEFILE, MAKE_FLAGS, MAKE_ENV. The following variables are specific for «install»: It is quite common to add post-install actions to Makefile, because often documentation files are not installed by the installation procedure defined by the packaged software. An example for such an action is:
post-install: 
    ${MKDIR} ${PREFIX}/doc/godi-getopt 
    ${CP} ${WRKSRC}/README ${WRKSRC}/COPYING  
      ${PREFIX}/doc/godi-getopt
There are a number of further variables here:

5.3.6  Dependencies

In Makefile one can also declare dependencies on other packages. The variables are: The syntax for the expressions one can use in these variables: where <BASENAME> is the name of the package without version string, <OPERATOR> is one of ==, >=, <=, >, <, !=, and <VERSION> is the version string the operator refers to. Note that package revisions in <VERSION> (i.e. any «godi» suffixes) are ignored, and one cannot enforce a certain revision.

The <PATH> is (almost) a legacy component of the dependency expressions, i.e. it is ignored by godi_console, but there are still some scripts that need it. The <PATH> must be set to
../../<CATEGORY>/<BASENAME>
where <CATEGORY> is the category prefix of <BASENAME> (apps, godi, etc.). The <PATH> is the relative path where to find the build directory of the package one refers to.

Examples for package dependencies: The comparison «>=0» should be read as «any version is accepted». There is also the legacy syntax «-*» or even «-[0-9]*» for the same purpose, e.g. «conf-zlib-*», but it should no longer be used in new packages.

Selecting the kind of dependency

BUILD_DEPENDS should only be used when it can be ensured that the referred package is only needed at build time, and in all other cases DEPENDS must be used. Examples when build-time dependencies are sufficient: In general, when a library is linked into a program, or a library is the antecedent for another library, the dependency is of the runtime type.

Magic dependencies

There are dependencies that need normally not to be listed, because it is impossible to even start the build process without a certain minimum equipment. These dependencies include: One can, however, demand certain minimum versions for these packages. This is sometimes useful when one needs a special feature that was only recently introduced into the packaging system.

5.3.7  Legacy expressions

In the past, it was necessary to put a number of expressions into Makefile to get a certain behaviour, but due to improvements this is no longer required: If you find that in existing packages, don't copy it to new packages.

5.4  Packing lists

The file PLIST describes the installed files, so GODI knows which files are owned by the package, and the files are removed when the package is deleted.

In general, the «install» stage must already have arranged that the files are installed in the right directories. Often, this is done by passing PREFIX to the configure script, but there is no general technique how this can be achieved. In doubt, read the documentation coming with the software, and, of course, the Makefiles and other build scripts included in the software distribution.

The PLIST file is needed first when the «install» stage is entered. Before this stage, the PLIST file can be omitted.

If you don't know the software you are packaging, it is sometimes difficult to find out which files are actually installed. As mentioned, the PLIST file is needed before the installation is done (at the beginning of the «install» stage), so you cannot just install and see what has been installed. See below for a discussion how to cope with this problem.

In the simplest form, PLIST just lists the files that have been installed, plus a number of directives how to deal with directories. For example:
bin/foo
This one-liner means that the package consists only of the executable bin/foo. Filenames are relative to the installation prefix of GODI, and it is an error to use absolute path names.

Because bin is a shared directory, no special handling of it is required.

For private directories, however, one should add @dirrm directives to PLIST. For example, this package installs files into lib/foo, and because this directory is considered to be the private property of the package, it must be removed when the package is deleted. This is achieved by the @dirrm directive:
lib/foo/bar.txt 
lib/foo/baz.a 
@dirrm lib/foo
As lib is again shared, it is not necessary to add another @dirrm directive for it. These directives should come after the files contained in the directories, and when private directories occur within private directories, the @dirrm directives must be in the order such that the inner directory comes first. (As you guess it, at package deletion time the PLIST is just interpreted line by line from top to bottom, and the removal actions are performed in this order.)

Note that the handling of directories is going to be changed. In the future, it will no longer be necessary to add @dirrm statements to PLIST. In a development version of godi_console, this revised handling is already implemented, but it will take some time to release it.

As it is quite error-prone to enumerate files, there are a number of abbreviations and special notations. We explain here only the most useful ones, for a complete reference see the file plist-ref.txt (usually installed in <PREFIX>/doc/GODI/project-doc). In most cases, these four directives are sufficient.

There is still the problem how to figure out which files are installed. A simple method:
  1. Begin with an empty PLIST, and do godi_make install
  2. Get the installed files by calling godi_make print-installed. Note, however, that there is no guarantee that this list is complete, the «print-installed» script checks the timestamps of all files in the GODI installation. This may go wrong, especially on systems where the «find» command does not support the comparison of the ctime field of the timestamp (which is less problematic than the mtime field).
  3. Write the PLIST according to the output of «print-installed». One should keep in mind that some files are only installed for certain system configurations, so just copying the list may not be enough.
  4. Now remove the package, and the files (so the package database is clean again):
    godi_delete <packagename>
    godi_make print-installed | ( cd <PREFIX>; xargs rm -f )
  5. Remove these files, and install the package again:
    rm work/.install_done work/.PLIST

5.5  An Example

Here the Makefile of godi-xstr (slightly updated):
.include "../../mk/bsd.prefs.mk" 
VERSION=        0.2.1 
PKGNAME=        godi-xstr-${VERSION} 
DISTNAME=       xstr 
DISTFILES=      xstr-${VERSION}.tar.gz 
CATEGORIES=     godi 
MASTER_SITES=   http://ocaml-programming.de/packages/ 
MAINTAINER=     gerd@gerd-stolpmann.de 
HOMEPAGE=       http://ocaml-programming.de 
COMMENT=        additional string functions

DEPENDS+=       godi-ocaml>=3.06:../../godi/godi-ocaml 
BUILD_DEPENDS+= godi-findlib>=0.8.1:../../godi/godi-findlib

MAKE_ENV+=      ${BUILD_OCAMLFIND_ENV}

USE_GMAKE=      yes

ALL_TARGET=     all 
.if ${GODI_HAVE_OCAMLOPT} == "yes" 
ALL_TARGET+=    opt 
.endif

post-install: 
        ${MKDIR} ${LOCALBASE}/doc/godi-xstr 
.       for F in README LICENSE 
            ${CP} ${WRKSRC}/${F} ${LOCALBASE}/doc/godi-xstr 
.       endfor

.include "../../mk/bsd.pkg.mk"
The corresponding PLIST file:
@findlib xstr  
doc/godi-xstr/README  
doc/godi-xstr/LICENSE  
@dirrm doc/godi-xstr

5.6  Further targets for godi_make

The following targets have relevance:

5.7  Testing packages

One of the basic properties every package must have is that one can delete it. So the most primitive test checks this:
  1. Install the package with godi_make install
  2. Delete the package with godi_delete, check that there are no error messages.
  3. Check that godi_make print-installed does not output anything!

5.8  The package repository

The package repository is realised with Subversion. Everybody can check it out:
svn checkout https://gps.dynxs.de/svn/godi-build
Of course, you need an account to modify any of the files. If you had one, you could do the following: I have prepared more detailed instructions how to perform these actions for everybody who wants to have an account on the GODI server. (You get these instructions together with the account data.)

5.9  Check list

(to be extended...)

Chapter 6  External Dependencies

The autoconf nightmare.

6.1  Configuration packages

6.2  The policy for libary lookup

6.3  Using godi_script to create configuration packages

This document was translated from LATEX by HEVEA.