Dr. Steven Simpson

Forge set-up

Passwords

…and other details to make note of, think about, or think up before starting.

adminPassword

This is the password of the privileged account you use to configure your system, the one you give when you sudo something.

mysqlRootPassword

You will be asked to specify this when the MySQL is set up. Use it with the username root. You'll need it to set up and maintain the database that holds courtesy accounts.

mysqlForgePassword

You'll need to specify a less privileged user to access the database routinely. For these instructions, the username forge is used with this password.

forgePassword

You'll need a normal user to own the repositories and the web server. For these instructions, the username forge is used with this password.

bindUserPassword

You'll need campus account to bind the LDAP client to AD, perhaps a service account. For these instructions, the username binduser is used with this password.

Database

The database holds details of the following:

• courtesy accounts
• hook scripts
• public keys, used to permit SSH access

Get MySQL installed:

sudo apt-get install mysql-server

As part of installation, this will require you to specify mysqlRootPassword, a root password for the MySQL database. Make a note of it.

(It's probably not a big deal to switch to PostgreSQL, if that's preferred. These instructions merely have the benefit of being known to work.)

Enter the database server to create a less privileged account, the database itself, and some tables:

sudo mysql --password

You might have to enter adminPassword first for the sake of sudo, but the prompt should be clear about this. Then you will definitely have to enter mysqlRootPassword, which you specified before.

Create a database forge which will hold details for courtesy accounts, hooks and public keys (and maybe later on, authorization details too):

CREATE DATABASE `forge` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
USE `forge` ;

Create a less privileged user forge within MySQL to access the forge database routinely:

CREATE USER 'forge'@'localhost' IDENTIFIED BY 'mysqlForgePassword';
GRANT LOCK TABLES , SELECT , INSERT , UPDATE , DELETE ON  `forge`.* TO 'forge'@'localhost';

You will need a table to describe courtesy accounts:

CREATE TABLE `forge`.`user_info` (
`user_name` VARCHAR( 40 ) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`user_passwd` VARCHAR( 13 ) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`user_fullname` VARCHAR( 60 ) CHARACTER SET utf8 COLLATE utf8_general_ci NULL ,
`user_email` VARCHAR( 60 ) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`user_affil` VARCHAR( 60 ) CHARACTER SET utf8 COLLATE utf8_general_ci NULL ,
`user_link` VARCHAR( 240 ) CHARACTER SET utf8 COLLATE utf8_general_ci NULL ,
PRIMARY KEY (user_name)) TYPE = MYISAM ;

You will need a table to hold public keys for SSH access:

CREATE TABLE forge.key_info (
`key_user` VARCHAR(40) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
`key_text` VARCHAR(1024) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
PRIMARY KEY (key_user)) TYPE = MYISAM CHARACTER SET utf8 COLLATE utf8_general_ci;

This is separate from the user table, because it includes users who access with their campus accounts too.

You will need a table to record hook scripts:

CREATE TABLE forge.`hook_info` (
`hook_id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`hook_repo` VARCHAR(20) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`hook_script` VARCHAR(30) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`hook_point` VARCHAR(20) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ,
`hook_active` BOOL NOT NULL DEFAULT '0',
`hook_arg1` VARCHAR(60) CHARACTER SET utf8 COLLATE utf8_general_ci NULL ,
`hook_arg2` INT NULL ,
`hook_arg3` VARCHAR(120) CHARACTER SET utf8 COLLATE utf8_general_ci NULL,
PRIMARY KEY (hook_id, hook_repo)
) TYPE = MYISAM CHARACTER SET utf8 COLLATE utf8_general_ci;

Database access examples

You might need to access the database manually, or write software to access it. Here are some example SQL statements to use.

User-table examples

List all users, first with all details, second with a selection:

SELECT * FROM forge.user_info ;
SELECT user_name, user_fullname FROM  forge.user_info ;

Create a new user called fred:

INSERT INTO user_info
   ( user_name, user_passwd, user_fullname,
     user_email, user_affil, user_link )
VALUES
   ( "fred", ENCRYPT("freds password"), "Fred Bloggs",
     "fred@bloggs.example.com", "Bloggs University",
     "http://www.fred.bloggs.example.com/" ) ;

Update part of fred's details:

UPDATE user_info SET user_affil = "Lancaster University"
 WHERE user_name = "fred" ;

Hook-table examples

A prepared statement to list script details for a given event on a repository:

SELECT * FROM forge.hook_info WHERE
( hook_repo = ? AND hook_active = TRUE AND hook_point = ? ) ;

Adding more columns when more sophisticated script types are added:

ALTER TABLE `hook_info` ADD `hook_point` VARCHAR( 20 )
CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL AFTER `hook_type` ;

HTTP server, SVN and other software

This should get the lot:

sudo apt-get install libaprutil1-dbd-mysql phpmyadmin libapache2-mod-php5 viewvc libaprutil1-ldap subversion libapache2-mod-python libapache2-svn tidy python-subversion ldap-utils build-essential openjdk-7-jdk gawk cifs-utils xsltproc

When asked about configuring phpmyadmin with dbconfig-common, say that you don't want to do anything yet.

Routine user account

Create a regular user to own repositories and the web server:

sudo adduser forge

Specify forgePassword as the password. You might have to specify the user id and group id to match the mount point that provides repositories via NFS, with --uid and --gid switches.

Create a place to store state (authorization, repositories), static resources (web pages), and configuration:

sudo mkdir -p /var/forge/{state/{svn-repos,authz},service/{docroot,cgi-bin,graphics,news,tmp,conf,scripts/hooks,hook-template}} /etc/forge/secure /usr/local/forge
sudo chown -R forge.forge /var/forge /usr/local/forge
sudo chmod 700 /etc/forge/secure

SVN+SSH account

This is hacky! Create an account which cannot be entered by password, but can by SSH:

sudo adduser --disabled-password --ingroup forge svn

Make this use the same id as forge, by carefully editing /etc/passwd, and changing ownership of its files:

sudo chown -R svn.forge ~svn
sudo -u forge mkdir --mode=700 ~svn/.ssh

Yeuch! There's possibly a better way, ensuring that svn belongs to the forge group, and ensuring umask is correctly set when accessing repositories. However, the forge account will still have to modify files belonging to svn, so perhaps that won't work.

Secure details

You need to create several files to hold vital credentials. Create blank files and give them restrictive permissions first:

sudo touch /etc/forge/secure/{dbd-cred.conf,ldap-cred-1.conf}
sudo chmod 600 /etc/forge/secure/{dbd-cred.conf,ldap-cred-1.conf}

Then provide the content:

# To go in /etc/forge/secure/dbd-cred.conf
DBDParams "dbname=forge user=forge pass=mysqlForgePassword sock=/var/run/mysqld/mysqld.sock"

# To go in /etc/forge/secure/ldap-cred-1.conf
AuthLDAPBindDN "CN=binduser,OU=...,DC=..."
AuthLDAPBindPassword bindUserPassword

You should be able to check this way:

ldapsearch -x -D 'CN=binduser,OU=...,DC=...' -W \
-H 'ldap://your.domain.controller.example:389' \
-b 'OU=...,DC=...' \
-s sub 'sAMAccountName=user'

If you haven't preserved the private key used for HTTPS, you'll have to create a new one:

sudo openssl genrsa -out /etc/forge/secure/privkey.pem 1024

If not already created, you will need to make an unsigned certificate request:

sudo openssl req -new \
   -key /etc/forge/secure/privkey.pem \
   -out /etc/forge/server.csr

To create or renew the self-signed certificate used here, use:

sudo openssl x509 -req -days 999 \
   -in /etc/forge/server.csr \
   -signkey /etc/forge/secure/privkey.pem \
   -out /etc/forge/server.crt

Apache configuration

In /etc/apache2/envvars, set the web server to run as forge instead of the default www-data:

export APACHE_RUN_USER=forge
export APACHE_RUN_GROUP=forge

SVN group authorization via Python

Get the Python script authz_svn_group.py, and put it in /usr/local/lib/python2.6/dist-packages/.

wget 'http://svn.apache.org/repos/asf/subversion/trunk/contrib/server-side/authz_svn_group.py' -O authz_svn_group.py
sudo cp authz_svn_group.py /usr/local/lib/python2.6/dist-packages/

Services

Enable built-in services, like HTTPS, DBD…:

sudo ln -s ../mods-available/ssl.conf /etc/apache2/mods-enabled/
sudo ln -s ../mods-available/ssl.load /etc/apache2/mods-enabled/
sudo ln -s ../mods-available/dbd.load /etc/apache2/mods-enabled/
sudo ln -s ../mods-available/authn_dbd.load /etc/apache2/mods-enabled/
sudo ln -s ../mods-available/ldap.load /etc/apache2/mods-enabled/
sudo ln -s ../mods-available/authnz_ldap.load /etc/apache2/mods-enabled/
sudo ln -s ../mods-available/headers.load /etc/apache2/mods-enabled/
sudo ln -s ../mods-available/authn_alias.load /etc/apache2/mods-enabled/

Create an authentication alias for MySQL:

# In /etc/forge/mysql.conf
DBDriver mysql
DBDPersist On
DBDMin  1
DBDKeep 2
DBDMax  10
DBDExptime 60
Include /etc/forge/secure/dbd-cred.conf

<AuthnProviderAlias dbd mysql>
  AuthDBDUserPWQuery "SELECT user_passwd FROM user_info WHERE user_name = %s"
</AuthnProviderAlias>

…and one for LDAP:

# In /etc/forge/ldap-1.conf
<AuthnProviderAlias ldap ldap-1>
  AuthLDAPURL ldap://your.domain.controller.example:389/OU=...,DC=...?sAMAccountname?sub?(objectClass=user)
  Include /etc/forge/secure/ldap-cred-1.conf
</AuthnProviderAlias>

…and enable them:

sudo ln -s /etc/forge/ldap-1.conf /etc/apache2/mods-enabled/forge-ldap-1.conf
sudo ln -s /etc/forge/mysql.conf /etc/apache2/mods-enabled/forge-mysql.conf

phpMyAdmin configuration

Enter database login details into /var/lib/phpmyadmin/config.inc.conf, to allow the database to be manipulated on-line. This makes sure that authentication is done by the HTTP server, rather than by phpMyAdmin itself:

<?php

$i=0;
$i++;
$cfg['Servers'][$i]['auth_type']     = 'config';
$cfg['Servers'][$i]['host']          = 'localhost';
$cfg['Servers'][$i]['user']          = 'root';
$cfg['Servers'][$i]['password']      = 'mysqlRootPassword';

?>

Make sure forge can read it:

sudo chown root.forge /var/lib/phpmyadmin/config.inc.php

Disable default exposure of phpMyAdmin, in /etc/phpmyadmin/apache.conf:

#Alias /phpmyadmin /usr/share/phpmyadmin

ViewVC configuration

Configure via /etc/viewvc/viewvc.conf:

#cvs_roots = ...
#svn_roots = ...
root_parents = /var/forge/state/svn-repos/ : svn
#default_root = ...
mime_types_file = /etc/mime.types
address = <a href="your contact URI">Your name</a>
languages = en-gb, en, fr, de, eo
#authorizer = svnauthz
#authzfile = /var/forge/state/authz/svn-access.conf

Enable the last two lines if you're using a version of ViewVC which supports authorization via SVN configuration. In fact, don't bother with linking it via CGI, unless you are…

Virtual host configuration

Create /etc/forge/vhost.conf:

<VirtualHost *:80>
  ServerAdmin admin@forge.example.com
  ServerName forge.example.com

  RedirectPermanent / https://forge.example.com/
</VirtualHost>

<VirtualHost *:443>
  ServerAdmin admin@forge.example.com
  ServerName forge.example.com
  ErrorLog /var/log/apache2/forge-error.log
  CustomLog /var/log/apache2/forge-access.log common

# Use SSL for this host.
  SSLEngine on
  SSLCertificateFile /etc/forge/server.crt
  SSLCertificateKeyFile /etc/forge/secure/privkey.pem
  SSLCipherSuite \
        ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
  <FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
  </FilesMatch>

# Provide the main content.
  DocumentRoot /var/forge/service/docroot/
  <Directory "/var/forge/service/docroot/">
    AllowOverride All
    Order allow,deny
    Allow from all
  </Directory>

# Add some static graphics.
  Alias /graphics/ "/var/forge/service/graphics/"
  <Directory "/var/forge/service/graphics/">
    Order allow,deny
    Allow from all
  </Directory>

# Add the generated news feeds.
  Alias /feeds/ "/var/forge/service/news/"
  <Directory "/var/forge/service/news/">
    Order allow,deny
    Allow from all
  </Directory>

# Set up the secure area of the site.
  <Location "/">
    AuthType Basic
    AuthName "Forge@Lancaster"
    AuthBasicAuthoritative off

    Require valid-user
    Satisfy any

    Order allow,deny
    Allow from all

    AuthBasicProvider mysql ldap-1
    AuthzLDAPAuthoritative off

    ErrorDocument 401 /errors/401
    ErrorDocument 403 /errors/403
  </Location>

  ScriptAlias /commands/ "/var/forge/service/cgi-bin/"
  <Location /commands/>
    Satisfy all
  </Location>

  <Directory "/var/forge/service/cgi-bin/">
    AllowOverride AuthConfig
  </Directory>

  <Location "/svn-repos/">
    DAV svn
    SVNParentPath /var/forge/state/svn-repos
    AuthzSVNAccessFile /var/forge/state/authz/svn-access.conf
    SVNIndexXSLT "/templates/svnindex.xsl"
  </Location>

# Set up a web interface to the local MySQL database.
  Alias /dbadmin/ "/usr/share/phpmyadmin/"
  <Location "/dbadmin/">
    DirectoryIndex index.php
    Options None
    Satisfy all
    Order allow,deny
    Allow from all

    PythonAuthzHandler authz_svn_group
    PythonOption AuthzSVNGroupFile \
                 /var/forge/state/authz/svn-access.conf
    PythonOption AuthzSVNGroupAuthoritative Yes

    Require group managers
  </Location>
</VirtualHost>

Enable this virtual host:

sudo ln -s /etc/forge/vhost.conf /etc/apache2/sites-enabled/050-forge

Create the file /etc/forge/profile.sh with this content:

export FORGE_STATE="/var/forge/state"
export FORGE_SERVICE="/var/forge/service"
export FORGE_SOFTWARE="/usr/local/forge"

export FORGE_HOST="forge.example.com"
export FORGE_SITE="https://${FORGE_HOST}/"

export FORGE_REPOS="${FORGE_STATE}/svn-repos"

export FORGE_WEBHOME="${FORGE_SERVICE}/docroot"

export FORGE_COMMANDPREFIX="/commands/"

export FORGE_SVNREPOSPREFIX="/svn-repos/"

export SVN_HOME="/usr"

Accessing MySQL from Java

Get mysql-connector-java-*.jar from MySQL Connector/J, and put it in /usr/local/forge/lib/. Create a symlink to it, mysql-connector-java.jar.

Mounting of repositories

Configure mounting of the repositories in /etc/fstab:

192.168.100.1:/store/forge /var/forge/state  nfs

Mount manually:

sudo mount /var/forge/state

Maintenance account

Log in as forge, and create ~/.bash_aliases:

#!/bin/bash

export http_proxy=proxy address
export MAKEFLAGS="-I${HOME}/.install/etc"

if [ -r ~/.install/etc/bash.pathcomp ] ; then
    . ~/.install/etc/bash.pathcomp
fi

prefix MANPATH /usr/share/man

prefix PATH ~/.install/bin
prefix MANPATH ~/.install/man

[ -r "/etc/forge/profile.sh" ] && . "/etc/forge/profile.sh"

Create /var/forge/service/conf/cgi-env.sh:

. /etc/forge/profile.sh

export SSWEBAUTH_CHARSET=UTF-8
export SSWEBAUTH_MULTIREPO=yes
export SSWEBAUTH_AUTHZFILE=${FORGE_STATE}/authz/svn-access.conf
export SSWEBAUTH_EXTAUTHZFILE=${FORGE_STATE}/authz/svn-access-ext.conf
export SSWEBAUTH_REPOHOME="${FORGE_REPOS}"
export SSWEBAUTH_REPOPREFIX="${FORGE_SVNREPOSPREFIX}"

export SSWEBAUTH_SWITCHUSERCMD="${FORGE_COMMANDPREFIX}switchuser"
export SSWEBAUTH_EDITSVNDIRCMD="${FORGE_COMMANDPREFIX}editsvndir"
export SSWEBAUTH_EDITUSERCMD="${FORGE_COMMANDPREFIX}useredit"
export SSWEBAUTH_EDITSVNGROUPCMD="${FORGE_COMMANDPREFIX}svngroupedit"
export SSWEBAUTH_NEWSVNREPOCMD="${FORGE_COMMANDPREFIX}createrepo"
export SSWEBAUTH_NEWUSERCMD="${FORGE_COMMANDPREFIX}adduser"
export SSWEBAUTH_USERLISTCMD="${FORGE_COMMANDPREFIX}listusers"
export SSWEBAUTH_ADMINREPOCMD="${FORGE_COMMANDPREFIX}repoadmin"
export SSWEBAUTH_EDITKEYCMD="${FORGE_COMMANDPREFIX}editkey"

export SSWEBAUTH_EDITSVNDIRSTYLE=/templates/editsvndir.xsl
export SSWEBAUTH_EDITUSERSTYLE=/templates/edituser.xsl
export SSWEBAUTH_NEWSVNREPOSTYLE=/templates/createsvnrepo.xsl
export SSWEBAUTH_NEWUSERSTYLE=/templates/adduser.xsl
export SSWEBAUTH_USERLISTSTYLE=/templates/userlist.xsl
export SSWEBAUTH_EDITSVNGROUPSTYLE=/templates/editsvngroup.xsl
export SSWEBAUTH_ADMINREPOSTYLE=/templates/adminrepo.xsl

export SSWEBAUTH_SVNHOME="${SVN_HOME}"

export SSWEBAUTH_MAKEREPO="${FORGE_SERVICE}/scripts/create-repo"
export HOOK_ROOT="${FORGE_SERVICE}/scripts/hooks"
export HOOK_TEMPLATE="${FORGE_SERVICE}/hook-template"

function sswebauth () {
  exec java \
    -classpath ${FORGE_SERVICE}/conf:${FORGE_SOFTWARE}/lib/sswebauth.jar:${FORGE_SOFTWARE}/lib/mail.jar:${FORGE_SOFTWARE}/lib/smtp.jar:${FORGE_SOFTWARE}/lib/mysql-connector-java.jar "$@"
}

Create /var/forge/service/conf/smtp.properties:

mail.host=your mail host
mail.smtp.host=your mail host
mail.from=the 'from' address in issued emails

Create /var/forge/service/conf/common-svnserve.conf:

[general]
authz-db = /var/forge/state/authz/svn-access.conf

Create /var/forge/service/conf/NewUserMail.properties, filling in the correct certificate fingerprints:

subject=Account created on Forge@Lancaster
body=\
                  DO NOT REPLY TO THIS MESSAGE!\n\
\n\
A 'courtesy' account was created for you on Forge@Lancaster at this\n\
location:\n\
\n\
  <https://forge.example.com/>\n\
\n\
The site uses a self-signed certificate with the following fingerprints:\n\
\n\
SHA1: XX:XX:XX\n\
\n\
MD5: XX:XX:XX\n\
\n\
Here are your credentials:\n\
\n\
        Username: {0}\n\
        Password: {1}\n\
\n\
Please log in and change your password as soon as possible via the web\n\
interface:\n\
\n\
  <https://forge.example.com/commands/useredit>\n\
\n\
That page also tells you which repositories you have access to.  To\n\
find out how to use them, see:\n\
\n\
  <https://forge.example.com/use>\n\
\n\
Please be especially aware of:\n\
\n\
  <https://forge.example.com/commit>\n\
\n\
\n\
Thankyou and Happy forging!

Create /var/forge/service/conf/mysql-creds.properties so that the admin programs can access courtesy accounts, etc:

touch /var/forge/service/conf/mysql-creds.properties
chmod 600 /var/forge/service/conf/mysql-creds.properties

username = forge
password = mysqlForgePassword

JavaMail installation

Install JavaMail's mail.jar and related libraries in /usr/local/forge/lib/mail.jar.

CGI scripts

Create a CGI script to run ViewVC in /var/forge/service/cgi-bin/svn-view:

#!/bin/sh

exec /usr/lib/cgi-bin/viewvc.cgi

Create /var/forge/service/cgi-bin/adduser:

#!/bin/bash

. ../conf/cgi-env.sh

sswebauth uk.ac.lancs.comp.webauth.CreateUser \
    -d com.mysql.jdbc.Driver \
    -s jdbc:mysql://localhost/ \
    -b forge \
    -t user_info \
    -c ${FORGE_SERVICE}/conf/mysql-creds.properties \
    -k user_name \
    -p user_passwd password ENCRYPT \
    -f user_fullname name \
    -f user_email email \
    -f user_affil affiliation \
    -f user_link website \
    -e email ${FORGE_SERVICE}/conf/smtp.properties NewUserMail \
    -m password \
    -m name \
    -m affiliation \
    -m website

Create /var/forge/service/cgi-bin/createrepo:

#!/bin/bash

. ../conf/cgi-env.sh

sswebauth uk.ac.lancs.comp.webauth.CreateSVNRepository

Create /var/forge/service/cgi-bin/editkey:

#!/bin/bash

. ../conf/cgi-env.sh

sswebauth uk.ac.lancs.comp.webauth.EditKey \
    -d com.mysql.jdbc.Driver \
    -s jdbc:mysql://localhost/ \
    -b forge \
    -t key_info \
    -c ${FORGE_SERVICE}/conf/mysql-creds.properties \
    -o /home/svn/.ssh/authorized_keys \
    -x "${SVN_HOME}/bin/svnserve" \
    -r "${FORGE_REPOS}/" \
    -fu key_user \
    -fk key_text

Create /var/forge/service/cgi-bin/editsvndir:

#!/bin/bash

. ../conf/cgi-env.sh

sswebauth uk.ac.lancs.comp.webauth.EditSVNFolder

Create /var/forge/service/cgi-bin/listusers:

#!/bin/bash

. ../conf/cgi-env.sh

sswebauth uk.ac.lancs.comp.webauth.ListUsers \
    -d com.mysql.jdbc.Driver \
    -s jdbc:mysql://localhost/ \
    -b forge \
    -t user_info \
    -c ${FORGE_SERVICE}/conf/mysql-creds.properties \
    -k user_name \
    -p user_passwd password ENCRYPT \
    -f user_fullname name \
    -f user_email email \
    -f user_affil affiliation \
    -f user_link website

Create /var/forge/service/cgi-bin/repoadmin:

#!/bin/bash

. ../conf/cgi-env.sh

sswebauth uk.ac.lancs.comp.webauth.AdminRepository \
    -d com.mysql.jdbc.Driver \
    -s jdbc:mysql://localhost/ \
    -c ${FORGE_SERVICE}/conf/mysql-creds.properties \
    -b forge \
    -t hook_info \
    -k hook_id \
    -Frepo hook_repo repo \
    -Fscript hook_script script \
    -Factive hook_active active \
    -Fpoint hook_point hook \
    -h email-on-commit post-commit hook_arg1 email hook_arg3 subject -- \
    -h email-on-commit-detailed post-commit hook_arg1 email hook_arg3 subject -- \
    -h email-on-commit-signal post-commit hook_arg1 email hook_arg3 subject hook_arg4 server -- \
    -h case-insensitive-uniqueness pre-commit -- \
    -h read-only start-commit -- \
    -h make-atom post-commit -- \
    -o read-only \
    -o make-atom \
    -o case-insensitive-uniqueness

# -d driver class to be loaded
# -s server URI
# -c credentials
# -b database name
# -t table name
# -k column (identity/key field)
# -Frepo column XML-attr (which repository)
# -Fscript column XML-attr (script name)
# -Factive column XML-attr (activation flag)
# -Fpoint XML-attr (hook type, e.g. post-commit)

# -k and -Frepo together define the table's key field, although hooks
# -for only one repo will be shown at a time, so -k is sufficient to
# -uniquely identify hooks in a repo.

# Additional parameters for each script
# -h script-name hook-type (column XML-attr)* --

Create /var/forge/service/cgi-bin/svngroupedit:

#!/bin/bash

. ../conf/cgi-env.sh

sswebauth uk.ac.lancs.comp.webauth.EditSVNGroup

Create /var/forge/service/cgi-bin/useredit:

#!/bin/bash

. ../conf/cgi-env.sh

sswebauth uk.ac.lancs.comp.webauth.EditUserDetails \
    -d com.mysql.jdbc.Driver \
    -s jdbc:mysql://localhost/ \
    -b forge \
    -t user_info \
    -c ${FORGE_SERVICE}/conf/mysql-creds.properties \
    -k user_name \
    -p user_passwd password ENCRYPT \
    -f user_fullname name \
    -f user_email email \
    -f user_affil affiliation \
    -f user_link website

Make them executable:

chmod 755 /var/forge/service/cgi-bin/{svn-view,adduser,createrepo,editkey,editsvndir,listusers,repoadmin,svngroupedit,useredit}

Repository creation

Create a script in /var/forge/service/scripts/create-repo to create a new repository using the hook framework HTTP authorization for its SSH authorization:

#!/bin/bash

# This shall be invoked with the name of a repository to be created.
# It will create one in the right directory, and set it up to use the
# database-configured hook scripts, and to use the same authorization
# for SSH as for HTTP.

repo="$1"

if [ "${0:0:1}" == "/" ] ; then
    here="$0"
else
    here="$PWD/$0"
fi
here="$(readlink -f "$here")"
here="${here%/*}"

. "${here}/../conf/cgi-env.sh"

${SVN_HOME}/bin/svnadmin create "$repo" && \
    mv "$repo/hooks" "$repo/hooks-orig" && \
    ln -s "${HOOK_TEMPLATE}" "$repo/hooks" && \
    mv "$repo/conf/svnserve.conf" "$repo/conf/svnserve.conf-orig" && \
    ln -s "${FORGE_SERVICE}/conf/common-svnserve.conf" "$repo/conf/svnserve.conf"

chmod 755 /var/forge/service/scripts/create-repo

Hook configuration

Create the generic script that works out which hook scripts to run on each repository by consulting the database, /var/forge/service/scripts/run-hooks:

#!/bin/bash

if [ "${0:0:1}" == "/" ] ; then
    here="$0"
else
    here="$PWD/$0"
fi
here="$(readlink -f "$here")"
here="${here%/*}"

. "${here}/../conf/cgi-env.sh"

hook_point="${0##*/}"

repo="${1##*/}"

export PATH=${SVN_HOME}/bin:${PATH}

function doit () {
    sswebauth uk.ac.lancs.comp.webauth.RunScripts \
        -d com.mysql.jdbc.Driver \
        -s jdbc:mysql://localhost/ \
        -c "${FORGE_SERVICE}/conf/mysql-creds.properties" \
        -b forge \
        -t hook_info \
        -k hook_id \
        -Frepo hook_repo repo \
        -Fscript hook_script script \
        -Factive hook_active active \
        -Fpoint hook_point hook \
        -R "$repo" \
        -P "$hook_point" \
        -h email-on-commit "${HOOK_ROOT}/email-on-commit" \
        arg 1 -+ \
        arg 2 -+ \
        col hook_arg1 -+ \
        col hook_arg3 -+ \
        txt dummy -+ \
        -- \
        -h email-on-commit-detailed "${HOOK_ROOT}/email-on-commit-detailed" \
        arg 1 -+ \
        arg 2 -+ \
        col hook_arg1 -+ \
        col hook_arg3 -+ \
        txt dummy -+ \
        -- \
        -h email-on-commit-signal "${HOOK_ROOT}/signal-on-commit" \
        arg 1 -+ \
        arg 2 -+ \
        col hook_arg1 -+ \
        col hook_arg3 -+ \
        col hook_arg4 -+ \
        -- \
        -h case-insensitive-uniqueness "${HOOK_ROOT}/case-insensitive-uniqueness" \
        arg 1 -+ \
        arg 2 -+ \
        arg 3 -+ \
        -- \
        -h make-atom "${HOOK_ROOT}/make-atom" \
        arg 1 -+ \
        arg 2 -+ \
        -- \
        -h read-only "${HOOK_ROOT}/read-only" \
        -- \
        "$@"
}

doit "$@"

Make it executable:

chmod 755 /var/forge/service/scripts/run-hooks

Link in the supported hooks:

ln -s ../scripts/run-hooks /var/forge/service/hook-template/start-commit
ln -s ../scripts/run-hooks /var/forge/service/hook-template/pre-commit
ln -s ../scripts/run-hooks /var/forge/service/hook-template/post-commit

Make sure that all SVN repositories use /var/forge/service/hook-template as their hook configuration:

for repo in /var/forge/state/svn-repos/* ; do
  mv "$repo/hooks" "$repo/hooks-orig"
  ln -s /var/forge/service/hook-template "$repo/hooks"
done

Case-insensitive uniqueness check

Create an executable hook script in /var/forge/service/scripts/hooks/case-insensitive-uniqueness to prevent commits that would result in some file names being distinct on by case. Use case-insensitive.py from the contrib directory in the Subversion source tar:

wget 'http://svn.apache.org/repos/asf/subversion/trunk/contrib/hook-scripts/case-insensitive.py' -O /var/forge/service/scripts/hooks/case-insensitive-uniqueness
chmod 755 /var/forge/service/scripts/hooks/case-insensitive-uniqueness

Atom feeds

Get the svn2feed.py script installed in /var/forge/service/scripts/:

wget 'http://svn.apache.org/repos/asf/subversion/trunk/tools/hook-scripts/svn2feed.py' -O /var/forge/service/scripts/svn2feed.py

Create a hook script in /var/forge/service/scripts/hooks/make-atom to build Atom feeds:

#!/bin/sh

repopath=${1}
revision=${2}

repopath=${repopath%%/}
reponame=${repopath##*/}


echo > /dev/stderr ${FORGE_SERVICE}/scripts/svn2feed.py -F atom \
    -f "${FORGE_SERVICE}/news/${reponame}.atom" \
    -r "$revision" \
    -u "${FORGE_SITE}commands/svn-view/$reponame" \
    -U "${FORGE_SITE}feeds/$reponame.atom" \
    -P ${SVN_HOME}/bin \
    "$repopath"

${FORGE_SERVICE}/scripts/svn2feed.py -F atom \
    -f "${FORGE_SERVICE}/news/${reponame}.atom" \
    -r "$revision" \
    -u "${FORGE_SITE}commands/svn-view/$reponame" \
    -U "${FORGE_SITE}feeds/$reponame.atom" \
    -P ${SVN_HOME}/bin \
    "$repopath"

Make them executable:

chmod 755 /var/forge/service/scripts/{svn2feed.py,hooks/make-atom}

Alternatively, as an executable in /var/forge/service/scripts/ss-svn-feed:

#!/bin/bash

repopath="${1}"
rev1="${2}"
diff="${3:-9}"
rev0="$((rev1 - diff))"
if [ "${rev0}" -lt 0 ] ; then rev0=0 ; fi

repopath="${repopath%%/}"
reponame="${repopath##*/}"

FEED_BASE="${FORGE_SITE}feeds/"
VIEW_BASE="${FORGE_SITE}commands/svn-view/"

title="$(svnlook propget "$repopath" repository-title / 2> /dev/null)"
if [ -z "$title" ] ; then
    title="$(svnlook propget "$repopath" repository-purpose / 2> /dev/null)"
fi
if [ -z "$title" ] ; then
    title="$reponame"
fi
website="$(svnlook propget "$repopath" repository-website / 2> /dev/null)"



function common_prefix () {
    local prefix line result prefix_top line_top

    read prefix

    while read line ; do
        result=""
        while true ; do
            prefix_top="${prefix%%/*}"
            prefix="${prefix#*/}"
            line_top="${line%%/*}"
            line="${line#*/}"
            if [[ "$prefix_top" != "$line_top" ]] ; then
                break;
            fi
            result="$result$prefix_top/"
        done
        prefix="$result"
    done

    echo "$prefix"
}


function xmlesc () {
    echo "$1" | \
        sed -e 's/&/\&amp;/g' -e 's/</\&lt;/g' -e 's/>/\&gt;/g' \
        -e 's/"/\&quot;/g' -e "s/'/\&apos;/g"
}



printf "<?xml version='1.0' ?>\n"
printf "<feed xmlns='http://www.w3.org/2005/Atom'>\n"
printf "<title>Commits on %s</title>\n" "$(xmlesc "$title")"
printf "<id>${FEED_BASE}%s.atom</id>\n" "$reponame"
printf "<link href='${FORGE_SITE}svn-repos/%s/' />\n" \
    "$reponame"
printf "<updated>%s</updated>\n" "$(date -u "+%Y-%m-%dT%T.%N%z")"

while [ "${rev1}" -ge "${rev0}" ] ; do

    log="$(svnlook log -r "$rev1" "$repopath")"
    author="$(svnlook author -r "$rev1" "$repopath")"


    prefix="$(svnlook dirs-changed -r "$rev1" "$repopath" | common_prefix)"

    when="$(svnlook date -r "$rev1" "$repopath")"
    when="$(date -d "$when" -u "+%Y-%m-%dT%T%z")"


    printf "<entry>\n"

    printf "<id>${FEED_BASE}%s.atom/%d</id>\n" \
        "$reponame" "$rev1"
    printf "<link href='${VIEW_BASE}%s?view=revision&amp;revision=%d' />\n" \
        "${reponame}" "${rev1}"

    printf "<title>r%d: /%s</title>\n" "$rev1" "$(xmlesc "$prefix")"


    printf "<summary>%s</summary>\n" "$(xmlesc "$log")"
    printf "<updated>%s</updated>\n" "$(xmlesc "$when")"
    printf "<author>%s</author>\n" "$(xmlesc "$author")"

    printf "</entry>\n"
    rev1="$((rev1 - 1))"
done

printf "</feed>\n"

make-atom then contains:

#!/bin/sh

repopath="${1}"
revision="${2}"

repopath="${repopath%%/}"
reponame="${repopath##*/}"

${FORGE_SERVICE}/scripts/ss-svn-feed "$1" "$2" | tidy -xml \
    > "${FORGE_SERVICE}/news/${reponame}.atom"

Emails on commit

Create a hook script in /var/forge/service/scripts/hooks/email-on-commit to send an email on each commit:

#!/bin/bash

function common_prefix () {
    local prefix line result prefix_top line_top

    read prefix

    while read line ; do
        result=""
        while true ; do
            prefix_top="${prefix%%/*}"
            prefix="${prefix#*/}"
            line_top="${line%%/*}"
            line="${line#*/}"
            if [[ "$prefix_top" != "$line_top" ]] ; then
                break;
            fi
            result="$result$prefix_top/"
        done
        prefix="$result"
    done

    echo "$prefix"
}


function doit () {
    prefix="$(${SVN_HOME}/bin/svnlook dirs-changed -r "$2" "$1" | common_prefix)"
    ${FORGE_SERVICE}/scripts/issue-svn-email "$prefix" "$@"
}

doit "$@"

/var/forge/service/scripts/hooks/email-on-commit-detailed is almost identical to email-on-commit, but issue-svn-email near the end needs to be issue-svn-email-detailed.

/var/forge/service/scripts/hooks/signal-on-commit is also almost identical to email-on-commit, but issue-svn-email near the end needs to be issue-svn-email-signal.

Make them executable:

chmod 755 /var/forge/service/scripts/hooks/{email-on-commit{,-detailed},signal-on-commit}

Read-only repositories

Create a hook script in /var/forge/service/scripts/hooks/read-only to prevent any commits:

#!/bin/bash

/usr/bin/printf "Repository is read-only\n"

exit 1

Make it executable:

chmod 755 /var/forge/service/scripts/hooks/read-only

Local package configuration

Create configuration in ~/.install/etc for several packages built locally:

# In ~/.install/etc/ss-scripts-env.mk
PREFIX=${HOME}/.install
CFLAGS += -g -O2
CFLAGS += -std=gnu99 -pedantic -Wall -W
CFLAGS += -Wno-unused-parameter
CPPFLAGS += -D_XOPEN_SOURCE=500

# In ~/.install/etc/svn-site-env.mk
WWWPREFIX=$(FORGE_WEBHOME)/

COMMAND_PREFIX=${FORGE_COMMANDPREFIX}
REPOS_ROOT=$(FORGE_REPOS)
REPOS_PREFIX=$(FORGE_SVNREPOSPREFIX)
NEWS_ROOT=$(FORGE_SERVICE)/news
NEWS_PREFIX=/feeds/

# Disable this to do some partial validation with HTMLTidy.                     
TIDY_XSLT2HTML=cat

OUTPUT=| $(HOME)/.install/bin/pipemv --tmpdir="$(FORGE_SERVICE)/tmp" "$@"

doit:: all

# In ~/.install/etc/svn-site-postenv.mk
HTML_LNK_SUFFIX=

# In ~/.install/etc/incontinent-env.mk
PREFIX=${FORGE_SERVICE}/software
CXXFLAGS += -std=gnu++98 -pedantic -Wall -W
CXXFLAGS += -g -O2

# In ~/.install/etc/sswebauth-env.mk
PREFIX=${FORGE_SOFTWARE}
CFLAGS += -std=gnu99 -g -O2
CXXFLAGS += -std=gnu++98 -g -O2

CPPFLAGS += -I${FORGE_SOFTWARE}/include
LDFLAGS += -L${FORGE_SOFTWARE}/lib

CLASSPATH += ${FORGE_SOFTWARE}/lib/mail.jar

PREFIX=${FORGE_SOFTWARE}
CXXFLAGS += -std=gnu++98 -pedantic -Wall -W
CXXFLAGS += -g -O2

# In ~/.install/etc/webscripts-env.mk
PREFIX=${HOME}/.install/opt/webscripts
HOOK_INCLUDE_PREFIX=${HOME}/.install/etc

# In ~/.install/etc/webscripts-local.mk
TIDY=tidy
M4=${HOME}/.install/opt/patched-m4/bin/m4
M4FLAGS += "-I$(HOME)/.install/include"
AWK=gawk

While keeping your log-in open, log in again to get another shell, and check that MAKEFLAGS is set.

Patched m4

Install the patched m4:

mkdir -p ~/incoming ~/scratch
wget 'http://www.comp.lancs.ac.uk/~ss/archives/m4-1.4.12-quotes-1.diff.bz2' -O ~/incoming/m4-1.4.12-quotes-1.diff.bz2
wget 'http://www.mirrorservice.org/sites/ftp.gnu.org/gnu/m4/m4-1.4.12.tar.bz2' -O ~/incoming/m4-1.4.12.tar.bz2
cd ~/scratch
tar xjf ~/incoming/m4-1.4.12.tar.bz2
bunzip2 < ~/incoming/m4-1.4.12-quotes-1.diff.bz2 | patch -p0
cd m4-1.4.12
./configure --prefix=$HOME/.install/opt/patched-m4
make && make install

Building local software

Create ~forge/.install/include/svn-site-user.m4 to tune the webpage output:

shift(


define(`UPDATED', `syscmd(`~/.install/bin/updated "+%Y-%m-%dT%H:%M:%SZ%z\n" "$1"')')

)dnl

Create a works directory to keep working copies:

mkdir ~/works
for i in misc/svn-site utils/incontinent utils/ss-scripts webtools/sswebauth webtools/webscripts ; \
  do svn checkout file:///var/forge/state/svn-repos/$i/trunk ~/works/${i#*/} ; \
done

cd ~/works/ss-scripts
make && make install

cd ~/works/incontinent
make && make install

cd ~/works/sswebauth
make && make install

cd ~/works/webscripts
make && make install

cd ~/works/svn-site
mkdir -p var/tables
make tables
touch var/newmonth
make

Icons

.htaccess

Create /var/forge/service/docroot/.htaccess:

Options +MultiViews +ExecCGI

AddLanguage en-GB .en-GB

ErrorDocument 401 /errors/401
ErrorDocument 403 /errors/403

AddType text/xsl .xsl

Header set opt "\"http://standard-sitemap.org/2007/ns\"; ns=15"
Header set 15-Location "/navigate.xml"

AddHandler send-as-is asis

Database back-up procedure

Create a private MySQL configuration file to hold credentials:

mkdir -p ~/.install/etc
touch ~/.install/etc/mysqldump.cnf

[mysqldump]
password=mysqlForgePassword

Perform daily:

mysqldump --skip-dump-date \
  --defaults-extra-file=$HOME/.install/etc/mysqldump.cnf \
  forge | bzip2 -9 > /backup/forge-0.sql.bz2

To restore:

bunzip2 < /backup/forge-0.sql.bz2 | mysql --password

Updated: 2011-Dec-20 13:06 GMT