Script-based Website Maintenance: Languages

Languages

Content-negotiated pages

Some webservers can be configured to present two or more versions of the same document under one address. The versions may differ in several ways, including the natural language in which the content is written.

Browsers can be configured to indicate which languages a user prefers to receive (consult your browser's documentation).

You can use m4 and make to generate documents in different langauges from the same source file. For example, from the file myfile.html.m4, you could generate myfile.html.en (the English version) and myfile.html.eo (the Esperanto version).

You need to indicate which languages you intend to work with in your Makefile:

AUTO_LANGUAGES=en eo

Use the ISO language codes.

You also need to tell your server to allow content negotiation on some files. Here's how to configure, for example, the Apache server to recognise the filename suffixes .en and .eo as indicating English and Esperanto versions of a page:

# in .htaccess
Options +MultiViews

AddLanguage eo .eo
AddLanguage en .en

Note:

You could probably get m4 to generate this automatically from KNOWN_LANGUAGES (that is my intention), but I haven't worked out the specifics yet.

:Note ends

Also give the stems of the source files for which language variants should be created:

HTML_DOCS=index files make parts/one parts/two parts/three

…i.e. create the files index.html.en, index.html.eo, files.html.en, files.html.eo…

The remaining definitions do not contain any site-specific text, so they should be put in your global Makefile, buildwww.mk.

These macros work out which language-dependent files should be created:

HTML_FILES+=$(foreach LANGUAGE, $(AUTO_LANGUAGES),\
$(HTML_DOCS:%=$(WWWPREFIX)%$(HTML_SUFFIX).$(LANGUAGE)))

XHTML_FILES+=$(foreach LANGUAGE, $(AUTO_LANGUAGES),\
$(XHTML_DOCS:%=$(WWWPREFIX)%$(XHTML_SUFFIX).$(LANGUAGE)))

SVG_FILES+=$(foreach LANGUAGE, $(AUTO_LANGUAGES),\
$(SVG_DOCS:%=$(WWWPREFIX)%$(SVG_SUFFIX).$(LANGUAGE)))

CSS_FILES+=$(foreach LANGUAGE, $(AUTO_LANGUAGES),\
$(CSS_DOCS:%=$(WWWPREFIX)%.$(CSS_SUFFIX).$(LANGUAGE)))

You also need to show how to build a language variant. This macro ensures that the m4 macro LANGUAGE is set to indicate which version is currently being created:

BUILDLANGFILE=$(BUILD) "-DLANGUAGE=\`$(subst .,,$(suffix $@))'" \
"-DFILE=\`$*$(suffix $(basename $@))'" $(SETUPDATED)

These commands build language-dependent files:

BUILD_LANG_HTML=@ \
	@echo Building $*$(HTML_SUFFIX)$(suffix $@) ; \
	export LC_ALL=$(subst .,,$(suffix $@)).UTF-8 ; \
	export SHELL=$(SHELL) ; \
	$(BUILDLANGFILE) -R $(SITE_HTML) $(LATEST_HTML) "$<" | \
		$(TIDY_HTML) $(OUTPUT)
BUILD_LANG_XHTML=@ \
	@echo Building $*$(XHTML_SUFFIX)$(suffix $@) ; \
	export LC_ALL=$(subst .,,$(suffix $@)).UTF-8 ; \
	export SHELL=$(SHELL) ; \
	$(BUILDLANGFILE) -R $(SITE_XHTML) $(LATEST_XHTML) "$<" | \
		$(TIDY_XHTML) $(OUTPUT)
BUILD_LANG_SVG=@ \
	@echo Building $*$(SVG_SUFFIX)$(suffix $@) ; \
	export LC_ALL=$(subst .,,$(suffix $@)).UTF-8 ; \
	export SHELL=$(SHELL) ; \
	$(BUILDLANGFILE) -R $(SITE_SVG) $(LATEST_SVG) "$<" | \
		$(TIDY_SVG) $(OUTPUT)
BUILD_LANG_CSS=@ \
	@echo Building $*$(CSS_SUFFIX)$(suffix $@) ; \
	export LANG=$(subst .,,$(suffix $@)) ; \
	export SHELL=$(SHELL) ; \
	$(BUILDLANGFILE) -R $(SITE_CSS) $(LATEST_CSS) "$<" | \
		$(TIDY_CSS) $(OUTPUT)

Finally, you need to express the basic dependencies for language-specific files:

var/langs.mk: Makefile
	@rm -f $@
	@mkdir -p `dirname $@`
	@for SRC in src var ; do \
	for TYPE in HTML XHTML SVG CSS ; do \
	for LANG in $(AUTO_LANGUAGES) ; do \
		printf >> $@ '$$(WWWPREFIX)%%$$(%s_SUFFIX).%s: ' \
			"$$TYPE" "$$LANG" ; \
		printf >> $@ '%s/www/%%$$(%s_SRC_SUFFIX).m4\n' \
			"$$SRC" "$$TYPE" ; \
		printf >> $@ '\t@mkdir -p `dirname $$@`' ; \
		printf >> $@ ' 2>/dev/null || true\n' ; \
		printf >> $@ '\t$$(BUILD_LANG_%s)\n' "$$TYPE" ; \
	done ; done ; done

…and include this in each site-specific Makefile:

sinclude var/langs.mk

Single-language generation

You may also want to build language-insensitive files from your language-sensitive source, having specified the required language. You need to list those files in buildwww.mk:

HTML_FILES_LANG=$(HTML_DOCS_ANY:%=$(WWWPREFIX)%$(HTML_SUFFIX))
XHTML_FILES_LANG=$(XHTML_DOCS_ANY:%=$(WWWPREFIX)%$(XHTML_SUFFIX))
SVG_FILES_LANG=$(SVG_DOCS_ANY:%=$(WWWPREFIX)%$(SVG_SUFFIX))
CSS_FILES_LANG=$(CSS_DOCS_ANY:%=$(WWWPREFIX)%$(CSS_SUFFIX))

HTML_FILES_LANG+=$(HTML_DOCS:%=$(WWWPREFIX)%$(HTML_SUFFIX))
XHTML_FILES_LANG+=$(XHTML_DOCS:%=$(WWWPREFIX)%$(XHTML_SUFFIX))
SVG_FILES_LANG+=$(SVG_DOCS:%=$(WWWPREFIX)%$(SVG_SUFFIX))
CSS_FILES_LANG+=$(CSS_DOCS:%=$(WWWPREFIX)%$(CSS_SUFFIX))

ALLFILES_LANG=$(HTML_FILES_LANG) $(XHTML_FILES_LANG) $(SVG_FILES_LANG) \
$(CSS_FILES_LANG)

…pass the language from the command line:

SETLANGUAGE="-DLANGUAGE=\`$(LANGUAGE)'"

…and, in the local Makefile, indicate what a language-insensitive site consists of:

all-lang:: $(ALLFILES_LANG)

$(HTML_FILES_LANG): $(SITE_HTML)
$(XHTML_FILES_LANG): $(SITE_XHTML)
$(SVG_FILES_LANG): $(SITE_SVG)
$(CSS_FILES_LANG): $(SITE_CSS)

To use all-lang, you must specify the LANGUAGE macro as well, e.g.:

$ make all-lang LANGUAGE=en

Language-dependent content

The m4 macro LANGUAGE tells us which language variant is being created, so, we can use it to select different translations of some text.

Each pair of arguments to this macro specifies a language and the content for that language; an additional argument is the default:

define(`LANG',`ifelse($#,0,,
$#,1,`$1',
LANGUAGE,$1,`$2',
`LANG(shift(shift($@)))')')dnl

This macro temporarily sets the language context (argument 1) for some content (argument 2):

define(`INLANG',`pushdef(`LANGUAGE',`$1')`'$2`'popdef(`LANGUAGE')')dnl

If you have installed my patch for m4, you can use a more sophisticated form of comparison. This macro expands to 1 only if its second argument encompasses its first:

define(`ISLANG',`ifelse(translit($2, `A-Z',`a-z', 1),
translit(ifelse(len($2),2,`substr($1,0,2,1)',`$1'), `A-Z',`a-z',1),1,0)')dnl

…so en encompasses en, en-us and en-gb.

Now you can change LANG to use it:

define(`LANG',`ifelse($#,0,,
$#,1,`$1',
ISLANG(`LANGUAGE',`$1'),1,`$2',
`LANG(shift(shift($@)))')')dnl

I keep these macros in a file include/ctxt.m4 which I include from etc/site.m4.

Quotation marks

Here are some language-dependent marks:

define(`Q_EN',`Qx(`$1',``&lsquo;'',``&rsquo;'',shift($@))')
define(`DQ_EN',`Qx(`$1',``&ldquo;'',``&rdquo;'',shift($@))')
define(`Q_EO',`Qx(`$1',``&rsaquo;'',``&lsaquo;'',shift($@))')
define(`DQ_EO',`Qx(`$1',``&raquo;'',``&laquo;'',shift($@))')
define(`Q_DE',`Qx(`$1',``&sbquo;'',``&lsquo;'',shift($@))')
define(`DQ_DE',`Qx(`$1',``&bdquo;'',``&ldquo;'',shift($@))')
define(`Q_FR',`Qx(`$1',``&lsaquo;'',``&rsaquo;'',shift($@))')
define(`DQ_FR',`Qx(`$1',``&laquo;'',``&raquo;'',shift($@))')

Here is a way to select different styles of quotation marks based on the current language in use:

define(`Q',`LANG(
``en'',`Q_EN($@)',
``fr'',`Q_FR($@)',
``de'',`Q_DE($@)',
``eo'',`Q_EO($@)',
`Qx(`$1',``?'',``?'',shift($@))')')
define(`DQ',`LANG(
``en'',`DQ_EN($@)',
``fr'',`DQ_FR($@)',
``de'',`DQ_DE($@)',
``eo'',`DQ_EO($@)',
`Qx(`$1',``??'',``??'',shift($@))')')

I keep these macros in a file include/xml.m4 which I include from etc/site.xml.m4.

Abbreviations

These are language-dependent abbreviations:

define(`IE',`LANG(``en'', ``i.e.'', ``t.e.'')')
define(`EG',`LANG(``en'', ``e.g.'', ``ekz.'')')
define(`ETC',`LANG(``en'', ``etc'', ``ktp'')')

Error documents

If someone visits your website, and doesn't understand any of the provided languages, he will receive an error document explaining this, but probably not in one of the languages he understands! You can override this document with something more meaningful, e.g., a single page with the same brief message in several languages.

Note:

More to follow…

:Note ends

When no suitable page exists, the browser receives an HTTP response with the code 406 (see 10.4.7 406 Not Acceptable). You need to tell the server to use your multilingual page with this code, e.g. for Apache:

ErrorDocument 406 /computing/users/ss/nolang.html

Note that Apache 2 already (partly) supports multi-language error documents.