RISC OS Shared Libraries

Shared, dynamically linked libraries for RISC OS, based on AOF

Files

Description

This is a collection of utilities for supporting shared, dynamically linked libraries on RISC OS. Hopefully, it meets some of the requirements of The One True Dynamic Linker, which is very informative about the issues.

The current status appears to be that I have a full, working system! However, it is very experimental, and still not thoroughly tested, so please try it out, and let me know how you fared. Remember to save any critical work before starting. It is probably too soon to start any practical development using this system at the moment, since several things have yet to be fixed — see the jobs below.

Note that this software is now probably obsoleted by ‘Experimental Dynamic Linking GCC for RISC OS’.

---

Components

The system consists of several, somewhat independent parts, described below. It should be possible to improve and replace them as people discover the shortfalls of my implementations.

Linking tools

roshar creates shared libraries by linking together several AOFs of re-entrant functions into a single AOF, including a ‘merged’ sb block, and vectors for non-leaf functions.

adlib adds an extra OBJ_LIBS chunk to an AOF file, which lists shared libraries on which the file depends. You can use it on the shared libraries created by roshar, or (more usually) to create AOF executables — use drlink or Acorn link to create a partially linked AOF from your program's static components, then use adlib to refer to the dynamic parts.

aofsld actually does this for you in one step. Specify the static and dynamic parts, and an AOF is produced containing the static parts and an OBJ_LIBS chunk refering to the dynamic parts.

Shared-library directory structure

!SharedLib is simply an application directory (analogous to !System), to be placed in !Boot.Resources, and contain commonly used shared libraries. It sets up a path, SharedLibraries:, through OS-dependent directories, so you can easily povide OS-dependent versions of a library in one distribution, to be merged with !SharedLib.

Library manager

LibraryManager is a kernel module to maintain shared libraries in memory, keeping track of dependencies and usage. In fact, it is a shared-AOF maintainer, identifying AOFs by case-insensitive name and timestamp. It will select areas that can be shared (such as those containing constants or re-entrant functions), and store a single copy in memory to be used multiply. Unshared areas are kept as an offset into the original AOF file, to be copied from the file once per instantiation (so the file must remain in place while the library is in use).

Whether an area is shared or not, all the associated symbols, strings and relocation directives are shared, so dynamic linking can be achieved with efficient memory use, even if full sharing cannot.

The module also provides a function to look up a library by a root name, a major version number, and a minimum minor number (as provided by an OBJ_LIBS chunk), and map them to the name of a corresponding AOF file (using the shared-library directory structure).

Shared code and data are kept in a dynamic area on 32-bit systems, or in the RMA otherwise.

AOF loader

The module AOFLoader defines a run-action for files of type &A0F, and assumes such files are AOFs. (Note that this does not mean that all AOF files should be typed this way, only those intended to be directly executed.)

The file, and any libraries specified by its OBJ_LIBS chunk, are installed using the library manager, then their areas are instantiated (unless already shared) and linked. The entry point is located (which could be in one of the dependent libraries) and is executed as an application.

Structures are set up as part of the program's image to ensure that the installed AOFs will be released after termination.

The linker defines the following symbols as the structure representing the refering code and the structure describing the whole image:

#include <kernel.h>
#include <riscos/shlibs/types.h>
#include <riscos/shlibs/rttypes.h>
extern struct _rt_use _rt_use_this;
extern struct _rt_root _rt_root_this;

SharedCLibrary stub

The standard C library should be available using this system, as an AOF. However, as an intermediate solution, a stub is provided to make use of the existing module SharedCLibrary. Since LibraryManager also uses that module, the solution may not be so intermediate!

---

Technical details

The OBJ_LIBS chunk

The tools adlib and aofsld can modify AOF files to insert an OBJ_LIBS chunk, that can indicate which libraries the file depends on. The library manager understands this chunk when it loads an AOF, and will ensure that those libraries are installed with the file. Together with the directory structure !SharedLib, it provides a mapping from library specifier to AOF file.

The OBJ_LIBS chunk is an array of 3-word elements. The first word of each element is an offset into the string table, specifying the name of the required library. The other two words are just unsigned integers, specifying the major and minor version numbers of the library. Incomplete elements at the end of the chunk should be ignored.

When searching for a library 〈name, a, b〉, LibraryManager tries to find a file SharedLibraries:name.a.c.d.AOF, where c ≥ b, and the greatest c is chosen, then the greatest d is chosen for that c. This allows a certain amount of version control.

Making libraries sharable

AOF files to be used as shared libraries should be compiled reentrantly (with the -apcs 3/reent switch on Norcroft). roshar can then combine several such AOFs into one, adding the necessary (unshared) vectors, for example:

v1      LDR     ip,base
        LDR     pc,f1
v2      LDR     ip,base
        LDR     pc,f2
v3      LDR     ip,base
        LDR     pc,f3
v4      LDR     ip,base
        LDR     pc,f4
base    DCD     new_static_base
f1      DCD     inter_link_addr_1
f2      DCD     inter_link_addr_2
f3      DCD     inter_link_addr_3
f4      DCD     inter_link_addr_4

This scheme has a limit of 512 vectors, but it is easily overcome by repeating the pattern for the next 512, etc.

roshar also changes B instructions so that the new program-counter value is loaded from the static-offset area. This helps the containing area to be shared, and allows the branch to reach beyond its 26-bit limit. For example, this:

        BL      printf

…becomes:

        BL      local
; at end of code area
local   LDR     pc,[sb,#offset]

; adcon area+offset
        DCD     printf

roshar merges the address-constant areas, removing duplicate words. The resultant area has a limit of 2047 words, which is achieved by making R9 point to the middle of the area, and using both negative and positive offsets. This needs testing!

Selection of shared areas

The library manager identifies areas with the following characteristics as ones which cannot be shared:

• it is not read-only,
• it is not position-independent (perhaps?),
• any of its relocation directives are symbol-based, or
• any of its relocation directives are based on an unshared area.

In fact, the library manager distinguishes between four modes of instantiation:

‘shared’

The area has been loaded into shared memory, and its relocation directives applied. It should be used directly.

‘resolved’

The area has been loaded into shared memory, and its relocation directives applied. It should be copied into application space for each instantiation.

‘in-memory’

The area has been loaded into shared memory. It should be copied into application space for each instantiation, and its relocation directives should be applied to the copy.

‘in-file’

The area has been left in the original AOF file. It should be loaded into application space for each instantiation, and its relocation directives should be applied to the copy.

The module identifies ‘resolved’ areas from the ‘shared’ ones by checking their symbols for function entry points — if there are any, and we're running on a 32-bit system, the area will have to be copied out to application space so that the branch instructions can reach it. Other shared libraries can always reach these entry points anyway, since they don't use branches to call them.

Handling shared leaf functions

Leaf functions don't have any special re-entrancy requirements, so vectors don't need to be created for them. Only PC needs to be set, and a branch from another shared library will already have been adjusted to load the new PC value from a word holding the function's address. From application space, the branch instruction itself must be adjusted at link-time to point to the function, but on a 32-bit system, the difference between the branch and the function may be too great to encode in the B instruction. Yet on a 26-bit system, we don't need to take any special action — how do we deal with these differences?

Currently, this is dealt with by the following steps:

• roshar generates a shareable area to hold simple vectors for any leaf functions it finds. Each vector looks like this:

          LDR     pc,[pc,#-4]
          DCD     real_entry_point
  

  The LDR instruction is given as the (false) entry point of the function.

• The library manager identifies the area as ‘shared’, and applies its relocation directives. But on a 32-bit system, its symbols won't be reachable from application space, and so the library manager sets the area to ‘resolved’.

• The linker sees that the area is already resolved (i.e. either ‘shared’ or ‘resolved’), so when it finds a symbol in the area, it can extract the true address, and use it if it has space for the 32-bit value. If not, it will use the vector as normal. Furthermore, on a 32-bit system, the vector area will be ‘resolved’, not ‘shared’, so the linker is expected to copy it out into application space, where a branch in the application can reach it.

Setting up the run-time environment

The usual stub for the module SharedCLibrary relies on the static linker providing it with some linker-defined symbols, to complete parts of the image and to indicate to the module where the program begins and ends in memory, so that only the remaining memory will be used for the heap and stack.

The dynamic linker doesn't define these all symbols, so a new stub is necessary. The linker does try to arrange the program image into two areas, so the following symbols are defined:

Image$$Base

Image$$Limit

These delimit the whole generated image in application space. They are the only ones used by the replacement SCL stub.

Image$$RO$$Base

Image$$RO$$Limit

The first part of the image consists of structures describing the image, and instantiations of the read-only areas.

Image$$RW$$Base

Image$$RW$$Limit

The second part of the image consists of the read-write and zero-initialised areas.

Image$$ZI$$Base

Image$$ZI$$Limit

This is an empty area, since all zero-initialised areas will have already been dealt with by the dynamic linker.

Image$$RO$$Base and Image$$Base are equal.

Image$$RO$$Limit and Image$$RW$$Base are equal.

Image$$RW$$Limit, Image$$ZI$$Base, Image$$ZI$$Limit and Image$$Limit are equal.

---

Usage

Installation

From roshlmgr.arc, merge the !System directory into your own, to install the library manager module.

From aofldr.arc, merge the !System directory into your own, to install the AOF loader.

From roshar.zip, copy the !Boot onto your own. This installs the three commands roshar, adlib and aofsld.

From roshlres.zip, copy !SharedLib to !Boot.Resources. Shared AOFs should be nested within here.

Create the directory !SharedLib.Default.scl.1.0.0. From roshlcstub, copy the file o.scl into it, under the name AOF. This command should do the work for you:

ShlInstall o.scl scl 1.0.0

Library creation

Any non-reentrant AOF can be used as a dynamically linked library — just place it somewhere in !SharedLib.Default, i.e. in the path SharedLibraries:.

To make a truly shared library (using Norcroft), generate at least one AOF file with re-entrant functions, and pass it (or them) through roshar. For example:

cc -c -apcs 3/32bit/reent c.part1 c.part2 c.part3
roshar -o o.lib o.part1 o.part2 o.part3

roshar preprocesses each input file through the normal linker, to help remove a few trivial relocation directives. With -p (the default), all the input files are partially linked into a single AOF, which roshar then processes. With +p, each file is partially linked separately, then roshar combines them itself. Use the -v switch to see what's going on.

roshar may not be complete — it may still leave a few relocation directives undone. Please check the output file for mistakes, and inform me if you find any.

Executable creation

Create your object files in the normal way (not re-entrantly), then link with (for example):

aofsld -o prog o.obj1 o.obj2 o.obj3 -sscl.1

This specifies that the three object files should be statically linked, and that scl version 1.0 or later should be dynamically linked. No check is made that all symbols will be resolved on execution.

prog will have the type &A0F.

Try a simple program that asks for some input:

#include <stdio.h>

int main(int argc, char **argv)
{
  char buf[100];
  int i;
  for (i = 0; i < argc; i++)
    printf("argv[%d] = \"%s\"\n", i, argv[i]);

  printf("Enter some text:\n");
  fgets(buf, sizeof buf, stdin);
  printf("You typed: %s", buf);
  return 0;
}

…and run it in a task window. When it's waiting for input, use LibraryManager_List to see what has been loaded. Then let the program finish, and see if the AOFs have been unloaded.

Execution

An application using shared libraries will need the following system initialisation:

| copied from http://www.iyonix.com/32bit/32bitIntro.shtml
RMEnsure UtilityModule 3.10 Error This application requires RISC OS 3.10 or later
RMEnsure UtilityModule 5.00 RMEnsure CallASWI 0.03 RMLoad System:Modules.CallASWI
RMEnsure UtilityModule 5.00 RMEnsure CallASWI 0.03 Error This app requires CallASWI 0.03 or later
RMEnsure FPEmulator 4.03 RMLoad System:Modules.FPEmulator
RMEnsure FPEmulator 4.03 Error This application requires FPEmulator 4.03 or later
RMEnsure SharedCLibrary 5.17 RMLoad System:Modules.CLib
RMEnsure SharedCLibrary 5.34 Error This application requires SharedCLibrary 5.34 or later

RMEnsure LibraryManager 0.00 RMLoad System:Modules.LibManager
RMEnsure LibraryManager 0.13 Error This application requires LibraryManager 0.13 or later
RMEnsure AOFLoader 0.00 RMLoad System:Modules.AOFLoader
RMEnsure AOFLoader 0.04 Error This application requires AOFLoader 0.04 or later

---

To-Dos

There are some outstanding issues to be discussed, below. Do you have pertinent experience or advice? Can you suggest any (better) solutions? If you have any valuable comments, please email them to me, or start a discussion on an appropriate newsgroup.

Reserve magic numbers

The SWI numbers are currently in the ‘user applications’ region, and the error numbers are totally arbitrary; the filetype &A0F has been appropriated — these should be properly allocated.

Optimistic library retention

Keeping unused libraries in memory in the expectation that they might be required again shortly may have some value. Or maybe not…

Choosing a single standard library

Object files compiled with different standard libraries may well not be compatible. If we don't select a single base library now, developers would have to provide two (or more) versions of each of their compiled libraries, and it would be a nightmare! This is true for any API with multiple competing implementations where part of an implementation is visible.

Integrate static linkers

By default, roshar passes all input files through link -aof, and that may not be able to handle a large resultant address-constant area. On the other hand, if roshar links them itself, it may miss some optimisations which increase the final size of the area. The ideal solution is that roshar and the conventional linker should be the same program.

However, I've just remembered that roshar itself generates many adcon words because of the branches, so the standard linker is much less likely to encounter a limit!

Version issues

Perhaps OBJ_LIBS should allow minimum revision numbers to be specified, but it would only be useful if the minor number matched exactly. This means adding an extra word to the entry format.

We might also want to incorporate the major version number into the library's name — it has to match exactly, just like the name, and it would save a directory level storing the library. I wanted to avoid this since it assumes long filenames. It would also mean removing a word from the OBJ_LIBS entry format.

---

Links

• The One True Dynamic Linker homepage

• RISC OS Shared Libraries - Drobe Launchpad

• RISC OS ELF GCC