PostgreSQL Installer build system 
=================================

This is the PostgreSQL Installer build system. This document attempts to 
describe how the system is architected, how to set it up and how to extend 
it. It is a work in progress and will no doubt require further refinement
over time. There there is one goal however:

Build all PostgresSQL & add-on package installers for all supported platforms
with a single command.

Note that this system is not intended to replace the existing installer system 
used on Windows (pgInstaller) - it is intended to mirror it's basic functionality
however.

Modular system design
---------------------

The modular system is designed to be as flexible as possible and allow package
authors as much freedom as possible in the way they design their installers. 
There are some basic rules about how we design add-on packages however - it 
remains up  to the individual author to determine whether or not breaking any 
rules will break their package. They had better not break the system though!

* Registration:

A central registry file is used in which packages should register themselves. This
data will be used by StackBuilder to locate installed packages. The registry 
file is /etc/postgres-reg.ini, and should be considered analagous in function to
the sections of the Windows registry used for the same purposes on that platform.

StackBuilder requires specific entries for the PostgreSQL server, as well as an
entry indicating the installed version of each unique package. An example file
is show below.

; This section is for a server, and is analagous to the PostgreSQL key under
; HKEY_CURRENT_USER\Software on Windows
[PostgreSQL\8.3]
Version=8.3.3
InstallationDirectory=/opt/PostgreSQL/8.3
DataDirectory=/opt/PostgreSQL/8.3/data
Port=5432
Superuser=postgres

[PostGIS_1_3_PG83]
Version=1.3.2

[pgAdmin3]
Version=1.8.4
InstallationDirectory=/opt/pgAdmin3

It is up to the uninstaller for each package to leave or clean the data during 
uninstallation. The version number for a package should *always* be
cleared, but other data may be retained. For example, the server package will
not remove the data directory, thus it is appropriate to leave the 
DataDirectory, Port and Superuser values intact.

* Installers:

Each package installer should be capable of being silently or interactively 
installing and uninstalling the package. When uninstalling, as much of the package
as possible should be removed, however it is not always possible (through
lack of reference counting between packages) or desirable to remove everything.

Build platform
--------------

The build platform is Mac OS X Snow Leopard. We use OS X because it allows us to run
all other Intel based OS's on the same machine. All other supported platforms
are run in VMWare Fusion virtual machines, with the exception of Solaris Sparc
which obviously must be an external machine. The supported platforms can be set
as external machines if required by exposing nfs shares from mac.

A number of additional dependencies are required when setting up the system,
such as libxml2 and libxslt, as well as useful utilties such as wget. Setting
these up correct is essentially to ensuring future Universal binary builds will
work as expected on all supported OS X versions (10.5/Leopard and above).

Utilities should be installed using MacPorts:

- Download the installer from http://www.macports.org/ and install the package.

- Add /opt/local/bin:/opt/local/sbin to the *end* of the path. Add the following
  line to ~/.bash_profile for the buildfarm user:

export PATH=$PATH:/opt/local/bin:/opt/local/sbin

- Install packages:

$ sudo port install cmake
$ sudo port install wget
$ sudo port install apache-ant
$ sudo port install bison
$ sudo port install flex
$ sudo port install ossp-uuid

Dependency libraries must be built with a little more control to ensure they 
use the correct SDK to allow them to be used on Tiger and above. We manually
build and install these packages into /usr/local/

- Download the source for each library (eg. libuuid, libxml2 & libxslt).

- Unpack the source into /usr/local/src

- Configure the source with a command such as:

CFLAGS="-isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 -arch i386 -arch x86_64 -arch ppc" LDFLAGS="-arch i386 -arch ppc -arch x86_64" ./configure --prefix=/usr/local/ --disable-dependency-tracking

- Build and install:

make all
sudo make install

Note that we must make sure all additional libraries link against these
libraries, and not the older, system copies. In the case of libxslt, we can
do this by configuring with --with-libxml-prefix=/usr/local

libuuid will typically link only in the primary architecture of the build
machine. To fix this, remove the library (libuuid.16.dylib) from .libs/ and
re-run the final link command, with the addition of appropriate -arch flags.

- Install unix2dos utility:
$sudo port install unix2dos

- Set up DocBook SGML

Configuring DocBook SGML on Mac OS X with MacPorts.

1) Install the following packages using MacPorts:

    opensp
    openjade
    docbook-xsl
    docbook2X

2) Download the docbook-dsssl stylesheets from 
http://sourceforge.net/project/showfiles.php?group_id=21935&package_id=16611 
and then unpack the archive in /opt/local/share/sgml

3) Add a symlink so PostgreSQL can find the stylesheets:

     sudo ln -s  /opt/local/share/sgml/docbook-dsssl-1.79 \
                 /usr/local/share/sgml/docbook-dsssl

3) Download DockBook 4.2 (http://www.docbook.org/sgml/4.2/docbook-4.2.zip) 
   and the ISO 8879 character entities (http://www.oasis-open.org/cover/ISOEnts.zip)

4) Unzip both archives into /opt/local/share/sgml/docbook-4.2

5) Ensure that all the unpacked files are world readable.

6) Run the following command in the docbook-4.2 directory:

    perl -pi -e 's/iso-(.*).gml/ISO\1/g' docbook.cat

7) Create the file /opt/local/share/sgml/catalog, with the following contents:

CATALOG "openjade/catalog"    
CATALOG "docbook-4.2/docbook.cat"
CATALOG "docbook-dsssl-1.79/catalog"

8) Make sure that settings.sh is updated with docbook installation path.

- Fix PAM headers

When building on Snow Leopard, using the Leopard SDK a failure may be seen when
configuring PostgreSQL for PAM. This is because the PAM header location moved 
from /usr/include/pam in 10.5 to /usr/include/security in 10.6. The difference between
the base OS and the SDK confuses configure. To avoid this, it may be necessary
to move the PAM headers in /usr/include/security/security so configure cannot find them.

- Downgrade Java

Snow Leopard only ships with Java 1.6, but most of our code, as well as pl/java
should be compiled with 1.5.

1) Download the Jlast Java update for Leopard: 
   http://support.apple.com/downloads/java_for_Mac_OS_X_10_5_Update_5
   
2) Download Pacifist from http://www.charlessoft.com/

3) * IMPORTANT * Delete the symlinks, 1.5 and 1.5.0 from 
   /System/Library/Frameworks/JavaVM.framework/Versions

4) Open the Java update in Pacifist, and extract the 
   /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0 directory into
   the corresponding location on the hard disk. 
   
5) Recreate the 1.5 symlink, to point to the 1.5.0 directory.

You can now use Java 1.5, by setting JAVA_HOME to: 
/System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Home

NOTE: That this update may have to be repeated following any Java updates from Apple.

Couple of packages needs hacking, while building on snow leopard.
PHP: http://www.firewing1.com/node/31
Slony-I: http://www.slony.info/bugzilla/show_bug.cgi?id=109


Build VMs
---------

All VMs (and in fact, the host machine) are setup to use user accounts called
'buildfarm'. In order to access each, the VMs must be setup with fixed IP
addresses which are recorded with an appropriate hostname in DNS. Each hostname 
is specified in settings.sh. It may be necessary to manually configure VMWare 
Fusion to bridge the network adaptor instead of using NAT.

The top level 'pginstaller' directory is shared with all the VMs using the VMware
shared folders feature. The path to this directory is specified in settings.h
for each VM. Note that VMware doesn't map UIDs/GIDs between the host and the VMs
so it may be necessary to mount the shared directory using the UID/GID of the
user in the VM, eg using the following in /etc/fstab:

.host:/  /mnt/hgfs  vmhgfs  defaults,ttl=5,uid=500,gid=500     0 0

SSH authentication between hosts is achieved using certificates. These can be
generated on the host machine using:

ssh-keygen -t rsa

Copy the resulting id_rsa.pub file to ~/.ssh/authorized_keys on each VM. 

* Linux/Linux-x64
- Install chrpath utility in order to change the rpath of the installed PostgreSQL binaries in the staging directory.
  Use the following command to install the chrpath:
  * yum install chrpath

* Windows

The Windows VM is the most tricky to setup:

- Install Windows 7
- Install Microsoft Sercurity Essentials
- Install Visual Studio 2008, and update to the latest service pack
- Create the 'buildfarm' user account, as a limited user.
- Install the a basic installation of Cygwin from http://www.cygwin.com/. 
  Include the OpenSSH package.
- Configure sshd with ssh-host-config, using the buildfarm user account as
  the service account. Make sure that openssh log file has correct ownership
  and permissions. It is absolotely necessary NOT to use default local account
  with sshd because compilation will fail when invoked via ssh.
- Make sure that port 22/TCP is open in the Windows Firewall configuration.
- Install the public ssh key in C:\Cygwin\home\buildfarm\.ssh
- Install zip.exe and unzip.exe into the System32 directory. These utilities
  can be found at ftp://ftp.tex.ac.uk/tex-archive/tools/zip/info-zip/WIN32/
- Create folder 'c:\pgBuild'
- Depending upon the modules to build, install various utilities in c:\pgBuild
- Install bison, flex, diffutils in c:\pgBuild (Available from http://gnuwin32.sourceforge.net)
- Prebuilt iconv, libxml2, libxslt, openssl and zlib from http://zlatkovic.com/pub/libxml,
  install them in c:\pgBuild
- gettext (Please consult developer for specific version for PostgreSQL)
- Mingw (gcc and g++). 
- MSys 1.0 
- Compile and install ZLib in Msys using --prefix=/mingw
- krb5 in c:\pgBuild
- vcredist in c:\pgBuild
- apache ANT in c:\pgBuild
- wxWidgets in c:\pgBuild

Note: In case of Windows-64 setup you may get errors related to: 
LINK : fatal error LNK1181: cannot open input file 'bufferoverflowU.lib'^M
Main reason of this issue is bufferoverflowU.lib file missing in parallel directory structures of VC installation. To reslove this copy said file into parallel structure e.g.:
cp /cygdrive/c/Program\ Files\ \(x86\)/Microsoft\ SDKs/Windows/v5.0/Lib/IA64/bufferoverflowu.lib  /cygdrive/c/Program\ Files/Microsoft\ SDKs/Windows/v6.0A/Lib/x64/.

* Mac OS X

Creating a new VM for new codepath from an existing VM on the same machine:
- Shutdown the VM
- Right click the VM and click 'show in finder' and then right click on the bundle to copy to another name
- Double Click the bundle to power it on and choose "I copied it" when Fusion asks
- Change the HostName, ComputerName using below commands:
  sudo scutil --set ComputerName "newname"
  sudo scutil --set LocalHostName "newname"
  sudo scutil --set HostName "newname"
  System Preferences->Users&Groups and Change full name to the new name
- Restart the VM
 
Build Machines as external machines
-----------------------------------
In order to set build machines as external machines, Create NFS share pointing to 
top level 'pginstaller' directory on Mac. For this purpose free tool 'NFS Manager' 
can be used. On linux side, update /etc/fstab to create nfs mount to this NFS share. 

Build scripts
-------------

* settings.sh

This script is derived from settings.sh.in which is stored in source control. It
is configured for the specific build machine, and allows us to specify what
platforms and modules we're building, and some global configuration options.

This script (_and_ the source version, settings.h.in) must be edited whenever
new platforms or packages are added.

* common.sh

This script contains common utility functions that may be used throughout the
build system.

* build.sh

This script is the main build script. To build everything, simply run the 
following command on the build host:

sh build.sh

For quick rebuilds, an option is provided to rebuild just the installers from 
the existing code in the staging directories:

sh build.sh -skipbuild

This script must be edited whenever a new module is added to call the appropriate
functions in the package build script.

Directories
-----------

* output/

This directory will contain all the completed installers.

* scripts/

This directory contains miscellaneous scripts that may be useful to multiple
modules or the overall build system.

* resources/

This directory contains installer resources that may be useful to multiple
modules or the overall build system.

* tarballs/

This directory contains all the tarballs we use for builds

* <everything else>/

Each additional directory contains a single package. These may be internally built
as required, though the interface should remain consistent - ie. a single build
script called build.sh, exposing functions called _prep_<packagename>, 
_build_< packagename > and _postprocess_< packagename >.

For a description fo the build system for a single package, see server/README.

Additional configuration in the VM's :
--------------------------------------

* Adding gd module to php in Windows
 
   * Prequisites:

     1) jpeg     (http://nchc.dl.sourceforge.net/sourceforge/gnuwin32/jpeg-6b-4.exe)
     2) libpng   (http://nchc.dl.sourceforge.net/sourceforge/gnuwin32/libpng-1.2.36-setup.exe)
     3) freetype (http://nchc.dl.sourceforge.net/sourceforge/gnuwin32/freetype-2.3.5-1-setup.exe)

     Install these in the pgBuild directory as jpeg, libpng and freetype respectively.

   * Modifications:

      Freetype:

       1) Modify the directory structure as:

          freetype --> include --> freetype2 --> freetype
          to
          freetype --> include --> freetype

          (leave the ft2build.h file in include directory as it is.)

       2) Copy the files:

          freetype/lib/freetype.lib to freetype/lib/freetype2.lib

      jpeg:

       1) Copy the files:

          jpeg/lib/jpeg.lib to jpeg/lib/libjpeg.lib


* Adding gd module to php in osx

   * Prequisites:
       1) Install jpeg libraries
       Download and extract jpeg from http://www.ijg.org/
       Compile and install: 
       >env CFLAGS="-isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 -arch i386 -arch ppc -arch x86_64" LDFLAGS="-arch i386 -arch ppc -arch x86_64" ./configure --prefix=/usr/local --disable-dependency-tracking
       >make
       >sudo make install
      
      2) Install libpng (User only 1.2.x version - php-5.2.1 has not yet include support for 1.4.x version)
      Download and extract libpng from http://www.libpng.org/pub/png/pngcode.html
      Compile and install:
      >env CFLAGS="-isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 -arch i386 -arch ppc -arch x86_64" LDFLAGS="-arch i386 -arch ppc -arch x86_64" ./configure --prefix=/usr/local --disable-dependency-tracking
      >make
      >sudo make install

      3) Install freetype
      Download and extract freetype from http://freetype.org/download.html
      >env CFLAGS="-isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 -arch i386 -arch ppc -arch x86_64" LDFLAGS="-arch i386 -arch ppc -arch x86_64" ./configure --prefix=/usr/local --disable-dependency-tracking
      >make
      >sudo make install

* Adding gd module to php in linux

   * Prequisites: (linux/linux-x64)

       1) yum install freetype
       2) yum install libpng
         (libjpeg.so should also be present in /usr/lib and /usr/lib64 for linux and linux-x64 respectively)

* Install the latest version of ActiveState Python, Perl & TCL/Tk on all
  the platforms.
* Install SPHINX for generating documentation for generating documentations for
  pgAdmin3.
  i.e. <PYTHONHOME>/bin/easy_install Sphinx
       For ActiveState Python 2.6, the PYTHONHOME is '/opt/ActivePython-2.6'
       For ActiveState Python 3.2, the PYTHONHOME is '/opt/ActivePython-3.2'
  NOTE: Install the SPHINX as the root user.

Trouble-Shooting:
* I got this error for ActivePython-3.2 on linux/linux-x64
    -----------------------------------------------------------------
      /opt/ActivePython-3.2/bin/python3.2
    ActivePython 3.2.2.3 (ActiveState Software Inc.) based on
    Python 3.2.2 (default, Sep  8 2011, 12:20:28) 
    [GCC 4.0.2 20051125 (Red Hat 4.0.2-8)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import hashlib;
    ERROR:root:code for hash md5 was not found.
    Traceback (most recent call last):
      File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 141, in <module>
        globals()[__func_name] = __get_hash(__func_name)
      File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 91, in
    __get_builtin_constructor
        raise ValueError('unsupported hash type %s' % name)
    ValueError: unsupported hash type md5
    ERROR:root:code for hash sha1 was not found.
    Traceback (most recent call last):
      File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 141, in <module>
        globals()[__func_name] = __get_hash(__func_name)
      File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 91, in
    __get_builtin_constructor
        raise ValueError('unsupported hash type %s' % name)
    ValueError: unsupported hash type sha1
    ERROR:root:code for hash sha224 was not found.
    Traceback (most recent call last):
      File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 141, in <module>
        globals()[__func_name] = __get_hash(__func_name)
      File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 91, in
    __get_builtin_constructor
        raise ValueError('unsupported hash type %s' % name)
    ValueError: unsupported hash type sha224
    ERROR:root:code for hash sha256 was not found.
    Traceback (most recent call last):
      File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 141, in <module>
        globals()[__func_name] = __get_hash(__func_name)
      File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 91, in
    __get_builtin_constructor
        raise ValueError('unsupported hash type %s' % name)
    -----------------------------------------------------------------
  In order to resolve the issue, I had to run the following command as 'root'
  user.
  chcon -t texrel_shlib_t /opt/ActivePython-3.2/lib/python3.2/lib-dynload/_hashlib.cpython-32m.so

Further info
------------

Contact [email protected] for further info.