Script-based Website Maintenance: Assembly rules

Assembly rules

You need to make two Makefiles, a global one and a site-specific one. You can do it all in one, but if you maintain several sites, a lot of the stuff here will be common to all of them. This way, you can add new features to all of them at once.

Global Makefile

Many of the rules you use to generate a website will not change for another site, so it makes sense to put these in a make include-file. Put them in the file include/buildwww.mk in your general directory.

The file should have the following parts:

These definitions express the structure of file names.

M4_SUFFIX=.m4
HTML_SUFFIX=.html
XHTML_SUFFIX=.xhtml
SVG_SUFFIX=.svg
CSS_SUFFIX=.css
HTML_SRC_SUFFIX=$(HTML_SUFFIX)
XHTML_SRC_SUFFIX=$(XHTML_SUFFIX)
SVG_SRC_SUFFIX=$(SVG_SUFFIX)
CSS_SRC_SUFFIX=$(CSS_SUFFIX)
INDEXBASE=index

You should be able to override these in your site-specific Makefile.

This macro accounts for language versions you want to create automatically, and other languages you recognise.
```
KNOWN_LANGUAGES=$(AUTO_LANGUAGES) $(OTHER_LANGUAGES)
```

These definitions show how to run various utilites.

SHELL=/bin/bash
M4=m4
M4FLAGS+=-I$(GLOBAL)/include -Iinclude
TIDY=tidy
TIDYFLAGS=-i -u -utf8 -q -numeric

HTML Tidy returns a non-zero value if there were any warnings or errors, but make will abort if a command returns non-zero. Yet we only want it to stop on error, so we need to modify its return value:
```
HTMLTIDY=$(GLOBAL)/bin/wraptidy $(TIDY) $(TIDYFLAGS)
```
Here's the script wraptidy:
```
#!/bin/sh -f

"$@"

result=$?

if [ $result -eq 2 ] ; then
  exit 1
fi

exit 0
```
If you don't want this behaviour, just use:
```
HTMLTIDY=$(TIDY) $(TIDYFLAGS)
```
It's not straight-forward to control charsets, because tools like tidy don't have fully independent control between input and output charsets, and don't recognise assigned charset names. However, this macro is passed through m4 to the pages. I'm still not sure how to make use of it properly.
```
CHARSET=UTF-8
```
Note:

A SRC_CHARSET would possibly be useful too.

:Note ends

You should instruct your server to add this charset to outgoing pages, e.g., for Apache:
```
# in .htaccess

AddDefaultCharset UTF-8
```
Note:

It should be fairly straight-forward to generate that file automatically, but I haven't worked out the specifics.

:Note ends
Set this to give the location of your global directories.
```
GLOBAL=$(HOME)/.install
```
You can override this to redirect the output of the scripts, which may be useful with CGI.
```
OUTPUT=> "$@"
```
These definitions show how to post-process different media types.
```
TIDY_HTML=$(HTMLTIDY)
TIDY_XHTML=$(HTMLTIDY) -xml
TIDY_SVG=$(HTMLTIDY) -xml
TIDY_CSS=awk -f $(GLOBAL)/etc/trimblanks.awk
```
Here is trimblanks.awk; it strips excessive blank lines (a basic kind of css-tidy!):
```
BEGIN { count = 1 }

/^$/ { count++;
       if (count == 1) print }

/^.+$/ { print ; count = 0 }

END { }
```
If tidy gives you error messages with line numbers, these won't correspond to your source files, but to the intermediate state between m4 and tidy. In this case, you can touch the errant source file, and rebuild with tidy disabled, using:
```
% touch src/www/thefile.html.m4
% make TIDY_HTML=cat
```
You'll have to preserve the error messages from the previous build, to compare their line numbers with the new generated file.

Note:

I wonder if one could use m4 to embed source line numbers in the HTML, so that tidy could directly give more useful error messages…

:Note ends

These macros compute the names of all the files that need to be produced. They use the macros HTML_DOCS_ANY, SVG_DOCS_ANY, and CSS_DOCS_ANY, which are provided by the site-specific Makefile.

HTML_FILES=$(HTML_DOCS_ANY:%=$(WWWPREFIX)%$(HTML_SUFFIX))
XHTML_FILES=$(XHTML_DOCS_ANY:%=$(WWWPREFIX)%$(XHTML_SUFFIX))
SVG_FILES=$(SVG_DOCS_ANY:%=$(WWWPREFIX)%$(SVG_SUFFIX))
CSS_FILES=$(CSS_DOCS_ANY:%=$(WWWPREFIX)%$(CSS_SUFFIX))
ALLFILES=$(HTML_FILES) $(XHTML_FILES) $(SVG_FILES) $(CSS_FILES)

These are the groups of site-specific files that each type depends on.

SITE_HTML_PRE=etc/site.m4 etc/site.xml.m4 etc/site.html-common.m4 \
etc/site.html.m4
SITE_XHTML_PRE=etc/site.m4 etc/site.xml.m4 etc/site.html-common.m4 \
etc/site.xhtml.m4
SITE_SVG_PRE=etc/site.m4 etc/site.xml.m4 etc/site.svg.m4
SITE_CSS_PRE=etc/site.m4 etc/site.css.m4

These are per-type intermediate ‘freezes’ of the above files.

SITE_HTML=var/site.html.m4f
SITE_XHTML=var/site.xhtml.m4f
SITE_SVG=var/site.svg.m4f
SITE_CSS=var/site.css.m4f

These are some more groups of site-specific files that affect each type. However, changing them does not cause a rebuild of all files of that type. Use these to experiment with new macros.

LATEST_HTML=etc/latest.xml.m4 etc/latest.html-common.m4 etc/latest.html.m4
LATEST_XHTML=etc/latest.xml.m4 etc/latest.html-common.m4 etc/latest.xhtml.m4
LATEST_SVG=etc/latest.xml.m4 etc/latest.svg.m4
LATEST_CSS=etc/latest.css.m4

These macros and rules build the freezes for each type. Note that many of the earlier make macros are converted to m4 macros.

PREPARE=$(M4) $(M4FLAGS) -F $@ \
	"-DINDEXBASE=\`$(INDEXBASE)'" \
	"-DHTML_SUFFIX=\`$(HTML_SUFFIX)'" \
	"-DXHTML_SUFFIX=\`$(XHTML_SUFFIX)'" \
	"-DSVG_SUFFIX=\`$(SVG_SUFFIX)'" \
	"-DCSS_SUFFIX=\`$(CSS_SUFFIX)'" \
	"-DHTML_SRC_SUFFIX=\`$(HTML_SRC_SUFFIX)'" \
	"-DXHTML_SRC_SUFFIX=\`$(XHTML_SRC_SUFFIX)'" \
	"-DSVG_SRC_SUFFIX=\`$(SVG_SRC_SUFFIX)'" \
	"-DCSS_SRC_SUFFIX=\`$(CSS_SRC_SUFFIX)'" \
	"-DWWWPREFIX=\`$(WWWPREFIX)'" \
	"-DKNOWN_LANGUAGES=$(shell \
		echo $(foreach lang, $(strip $(KNOWN_LANGUAGES)), \
		\\\`\\\`$(lang)\'\') | tr ' ' ',')" \
	"-DAUTO_LANGUAGES=$(shell \
		echo $(foreach lang, $(strip $(AUTO_LANGUAGES)), \
		\\\`\\\`$(lang)\'\') | tr ' ' ',')" \
	"-DCHARSET=\`$(CHARSET)'" > /dev/null

PREPARE_HTML=$(PREPARE) $(SITE_HTML_PRE)
PREPARE_XHTML=$(PREPARE) $(SITE_XHTML_PRE)
PREPARE_SVG=$(PREPARE) $(SITE_SVG_PRE)
PREPARE_CSS=$(PREPARE) $(SITE_CSS_PRE)

$(SITE_HTML): $(SITE_HTML_PRE)
	@echo Building HTML freeze
	@$(PREPARE_HTML)

$(SITE_HTML): $(SITE_XHTML_PRE)
	@echo Building XHTML freeze
	@$(PREPARE_XHTML)

$(SITE_SVG): $(SITE_SVG_PRE)
	@echo Building SVG freeze
	@$(PREPARE_SVG)

$(SITE_CSS): $(SITE_CSS_PRE)
	@echo Building CSS freeze
	@$(PREPARE_CSS)

These commands process a file through m4. Extra information like timestamps are added.
```
BUILD=$(M4) $(M4FLAGS)

BUILDFILE=$(BUILD) \
$(SETFILE) \
$(SETLANGUAGE) \
"-DLANGUAGES=ONLYLANGS(\`KNOWN_LANGUAGES', \
\`FILE_LEAF(\`\`$@'')')" $(SETUPDATED)
```
You need to indicate how to determine which file is being constructed, relative to the top of your website. LLINK uses this:
```
SETFILE="-DFILE=\`$*$(suffix $@)'"
```
This alternative strips out language suffixes, if you need that sort of thing:
```
SETFILE="-DFILE=FILE_PARENT(\`\`$*'')\`'NOLANGS(\`KNOWN_LANGUAGES', \
\`FILE_LEAF(\`\`$@'')')"
```
Set this macro to run a command to provide a last-modified time of the source file:
```
SETUPDATED="-DFILEUPDATED=\`$(shell updated +%Y-%b-%d\ %R\ %Z '$<')'"
```

These commands build language-independent files.

BUILD_HTML=@ \
	@echo Building $*$(HTML_SUFFIX) ; \
	$(BUILDFILE) -R $(SITE_HTML) $(LATEST_HTML) "$<" | \
		$(TIDY_HTML) $(OUTPUT)
BUILD_XHTML=@ \
	@echo Building $*$(XHTML_SUFFIX) ; \
	$(BUILDFILE) -R $(SITE_XHTML) $(LATEST_XHTML) "$<" | \
		$(TIDY_XHTML) $(OUTPUT)
BUILD_SVG=@ \
	@echo Building $*$(SVG_SUFFIX) ; \
	$(BUILDFILE) -R $(SITE_SVG) $(LATEST_SVG) "$<" | \
		$(TIDY_SVG) $(OUTPUT)
BUILD_CSS=@ \
	@echo Building $*$(CSS_SUFFIX) ; \
	$(BUILDFILE) -R $(SITE_CSS) $(LATEST_CSS) "$<" | \
		$(TIDY_CSS) $(OUTPUT)

This rule builds additional rules that cannot be easily expressed otherwise in a Makefile.

var/types.mk: Makefile
	@rm -f $@
	@mkdir -p `dirname $@`
	@for SRC in src var ; do \
	for TYPE in HTML XHTML SVG CSS ; do \
		printf >> $@ '$$(WWWPREFIX)%%$$(%s_SUFFIX): ' "$$TYPE" ; \
		printf >> $@ '%s/www/%%$$(%s_SRC_SUFFIX).m4\n' \
			"$$SRC" "$$TYPE" ; \
		printf >> $@ '\t@mkdir -p `dirname $$@`' ; \
		printf >> $@ ' 2>/dev/null || true\n' ; \
		printf >> $@ '\t$$(BUILD_%s)\n' "$$TYPE" ; \
	done ; done

These rules clear up the mess!

tidy:
	find . -name "*~" -exec rm -f "{}" \;

clean: tidy
	rm -rf var

Site-specific Makefile

Create a file Makefile in your site-specific directory. It should consist of:

an indication of the default rule, an inclusion of the global Makefile, and definitions of the environment that the global Makefile uses (possibly overriding some defaults);
```
all::

include buildwww.mk

WWWPREFIX=$(HOME)/www/pub/websitemgmt/
```

an indication of which files should be created;

HTML_DOCS_ANY=index files make parts/one parts/two parts/three
XHTML_DOCS_ANY=xdoc1 my/xdoc2
SVG_DOCS_ANY=pic1 pic2
CSS_DOCS_ANY=sty1 sty2

an inclusion of a generated Makefile to contain rules too complicated to be expressed otherwise (Make creates this automatically if it does not exist.);
```
sinclude var/types.mk
```
an indication of all files to be built (This is computed automatically from the earlier macros.);
```
all:: $(ALLFILES)
```

dependencies for all files and files of a certain type;

$(HTML_FILES): $(SITE_HTML)
$(XHTML_FILES): $(SITE_XHTML)
$(SVG_FILES): $(SITE_SVG)
$(CSS_FILES): $(SITE_CSS)

anything else specific to your website.

Building the files

To build (or rebuild) the files of your website, go into the directory containing its makefile, and type:

$ make

This will build the required files in $(WWWPREFIX) that do not already exist, or are not up-to-date with respect to their sources. Intermediate files are created in var.