ARIADNE Project on Digital Libraries · Publications

Information retrieval systems should acknowledge the existence of collaboration in the search process. Collaboration can help users to be more effective in both learning systems and in using them. We consider some issues of viewing interfaces to information retrieval systems as collaborative notations and how to build systems that more actively support collaboration. We describe a system that embodies just one kind of explicit support; a graphical representation of the search process that can be manipulated and discussed by the users. By acknowledging the importance of other people in the search process, we can develop systems that not only improve help-giving by people but which can lead to a more robust search activity, more able to cope with, and indeed exploit, the failures of any intelligent agents used.

Keywords

Process Visualisation, Information Retrieval, Interface Design, Collaborative Learning

… support for communication and collaboration is as important as support for information-seeking activities and that, indeed, support for the former is needed to support the latter.

(Levy and Marshall, 1995)

This social approach, taking direct advantage of human intermediaries' knowledge, in addition to any sophisticated underlying functionality, such as intelligent agents, is clearly more robust and adaptable than solely constructing an 'intelligent' stand-alone interface. It should be generalisable across databases, reactive to system change and likely to become more common as asynchronous interactions dominate accesses to remote digital libraries (Twidale, Nichols and Paice, to appear). We also claim that this approach, and systems that take account of it, fit better with the way that many people work (Grosser, 1991). Information searching is part of people's larger work activities which generally involve some interaction with colleagues. These interactions can include recommendations of relevant items, the sharing of search tactics, and informal explanations and help about how to use a particular system. Similar activity has been observed in a range of contexts including the use of spreadsheets (Nardi and Miller, 1991) . Systems which acknowledge the existence of formal and informal collaborations, and actively support it are likely to prove more useful and usable than current systems which seem to have been designed on the assumption of usage solely by individuals and yet which are often used in collaborative contexts.

The reasons why users are likely to need help are clear: the quantity of information is growing rapidly, the variety of information forms available electronically is also increasing, and there is a rapid rate of change of new information systems, new functionalities and new interfaces to existing systems. Consequently even the most enthusiastic user of information searching technology is faced with a continual need to update her skills. Furthermore we can assume that many (perhaps most) users do not find the technology intrinsically interesting but rather a tedious means to an end of obtaining the information that they are interested in. Thus we can expect the problem of the continued difficulties that users have with online public access catalogues (OPACs) (Borgman, 1996) despite the improvement in their interface design to persist. We need to design systems that are usable by Borgman's 'perpetual novices'.

We are proposing explicit support for collaboration rather than automation as a way of addressing user's difficulties. However, collaboration has its costs; someone has to pay for the interaction with an expert user, and with large numbers of searchers, help from an expert may be hard to get. Thus we must be concerned with supporting cost-effective collaboration. This includes enabling users to be able to quickly explain to each other what they have already managed to do and what they now need help on. It will involve minimising the amount of the expert's time that a help session will take by, for example, giving the expert a better understanding of the context of the query and making the delivery of help more efficient. In addition, we wish to investigate how users can minimise their recourse to experts by facilitating the incremental learning of search skills, and by learning from their peers.

In this paper we consider some issues of viewing interfaces as collaborative notations and how to build systems that more actively support formal and informal collaboration. We describe our first approach to implementing a system that embodies just one kind of explicit support for collaboration; a graphical representation of the search process.

Collaboration in Information Retrieval

Collaboration already occurs in current physical libraries. However, in addition to the much studied reference interview with a skilled intermediary, substantial informal collaboration occurs. We have observed that many students at Lancaster University Library learn how to use the OPAC from their peers, leaning across adjacent terminals or clustering around a single terminal, engaged in synchronous co-located collaborative learning and working (Twidale, Nichols and Paice, to appear). It is noteworthy that this collaboration occurs despite rather than because of the systems, which seem to be designed on the assumption that users work alone. The collaboration is also desirable, leading to effective learning and reducing the load on library staff.

The nature of collaborative IR activities will change with the trend towards remote digital libraries, and interfaces should support both existing (under-acknowledged) forms of collaboration and the new kinds of collaborative activities opened up by networking. We are studying the nature of collaboration as it currently occurs in conventional libraries to give us a starting point for developing more advanced systems for use in Digital Libraries. To date we have mostly looked at the collaboration that occurs between users, chiefly because this has in the past received far less attention than the expert-novice style interaction of the conventional reference interview. (Fowell and Levy, 1995) discuss issues of remote help-giving, or more generally 'networked learner support'; examining how the working practices of professional librarians will need to adapt to the new environment.

Interface Design

Building on information gleaned from the literature and interviews with subject librarians and a small sample of users, we created an exploratory environment to study collaborative browsing by combining existing computational tools. The aim was to undertake a rapid series of studies whose results would inform the design of our subsequent system. Consequently it was important to be able to rapidly create a number of experimental environments using technology to hand. We were particularly keen at this stage to investigate potential interactions between a relative novice and a more expert user such as a subject librarian. Volunteers were asked to bring along a genuine information need and try and address it in our laboratory setup.

We undertook a series of preliminary studies (Twidale, Nichols and Paice, to appear) of synchronous remote help giving using a combination of rudimentary tools including Unix talk. It became clear that a crucial need was to provide greater context by giving users the ability to share not only the search product (the hits obtained) but also the search process (how those hits were obtained). We have recently studied interactions between users and library staff at a help desk using ethnographic techniques (Crabtree et al, 1997). This study reinforced the importance of context; library staff would almost snatch at any piece of paper that an enquirer was holding (such as a book list, coursework specification or rough search notes) as a way of enriching the context of the articulated problem.

Not surprisingly when confronted by users claiming to be unable to find something, librarians frequently ask what they have tried. Equally unsurprisingly, users find this a difficult question to answer: it is very difficult to remember the search actions when one is focusing on the search goals, and there is a natural human inclination to auto-correct, remembering what one intended to type rather than what one did. Even worse, those most often asking for help are usually the ones who understand the system least and so are most likely to get confused and to lack a specialist vocabulary for describing their actions. By this argument we can see the desirability of recording the search activities so that in response to the question "What did you do?", the user can hand over the record and say: "This".

Thus even in a face-to-face help interaction, it is useful to have an interface that enables participants to share a representation of the user's search, and indeed perhaps of an expert's proposed solution. Where the user and intermediary are separated by distance and time, such an interface becomes even more important. Our study also emphasised the point that information seeking is a multi-stage process (Bates, 1989). It does not consist of just formulating the 'right' query expression and getting back what you want. Often it is only by the forming of a sequence of search expressions and reading the partial results that users are able to decide (or at least articulate) what it is that they actually want. The goals of a search may evolve based on the results of previous search actions. A search activity can consist of the composition of many queries and can involve leaving a terminal, to consult books or people, and may include breaks while other work is done, so that a search may extend over an arbitrarily long period of time, not just the few seconds of composing a query and reading its results. Our systems should acknowledge this and more actively support it. Sadly this knowledge has been available (Bates, 1989), but seems to have been rarely addressed in designing systems and their interfaces. However some systems do provide a few features to support it. For example BIDS keeps a record of the search queries issued that can be referred to (and replayed) within a session. Mackinlay et al.'s innovative interface (Mackinlay, Rao and Card, 1995), while focusing on the product of searches, does actively support the searching process seen as a larger activity, particularly the technique of citation chaining.

One method to achieve commonality is to use a video-link between users - typically between end-user and library staff . However, video is not sufficient for many collaborative activities as it fails to make the search a discrete manipulable object (Sugimoto et al, 1995). The C-TORI model of (synchronous or near-synchronous) cooperative IR provides facilities for sharing query histories, cooperatively browsing results and cooperative query formulation (Hoppe and Zhao, 1994). Nevertheless, the costs and inconvenience of synchronising help or collaboration are likely to override the advantages of its immediacy.

The search needs to be preserved and turned into a visualisation in order to support communication between people. However almost as a side effect, the resulting visualisation may be of use to a sole user. In this case, activities to support social use coincide with immediate personal benefit to the user. As a result we may not encounter the problem of a mismatch between those who have to put extra effort into and those who benefit from a collaborative system (Grudin, 1994).

Once a system supports the creation of such an interaction history, it can used in many ways (Hoppe and Zhao, 1994). Single-user uses include: reuse, error recovery, navigation, reminding and user modelling (Lee, 1992). Providing a complete record of a search activity frees users from having to remember low-level goalstack details and enables them to concentrate on more strategic elements of the search process. As a digital object it can also be communicated to other people - serving as the basis for a much wider range of collaborative activities (Twidale, Nichols and Paice, to appear). Such a search object can be stored, highlighted, edited, annotated, replayed and re-executed: typically the kinds of activities that occur in tutorial and help-giving interactions (Lemaire and Moore, 1994).

The Ariadne Interface

We have developed a prototype interface, Ariadne, which attempts to support these activities. Ariadne runs on top of an OPAC or online database, recording the queries and their results and then subsequently producing a visualisation of the search process that can be reflected on, shared and discussed by interested parties. The recording of the information occurs behind the scenes, the user interacts with the database in the normal way.

In part the design of the interface has been inspired by work on interfaces for tutoring systems drawing on the AI and Education paradigm, particularly regarding the explicit representation of processes and abstract concepts (Collins and Brown, 1988). The visualisation of the search process provided by Ariadne is a means for searchers to reflect on their activities. Reflection in information searching, as in other domains, is facilitated by making explicit 'the student's problem-solving processes - their thrashings, false starts and restarts, and partial successes' (Collins and Brown, 1988). Access to a representation of the intermediate states of the search process, such as in Ariadne, is crucial for effective post-problem reflection (Foss, 1987; Katz and Lesgold, 1994). Similarly, a computational representation can make visible aspects that are internalised to an expert but are difficult for a novice to acquire precisely because of their conventionally invisible nature (Hollan, Hutchins and Weitzman, 1984). The reification process offers great advantages in facilitating both learning about the concept and cooperative working using the concept.

A search is represented as a sequence of cards each containing the input search command (at the top) and the output result screen in thumbnail outline below. Cards can be expanded to a readable size by clicking on them. In Figure 1 a card has been clicked on and its expansion is visible. This reveals it to be part of a search of Lancaster's OPAC. The user has undertaken a title search on the word 'autism' and after scrolling through the list of hits has chosen to look at the record of a particular hit. By looking at the unexpanded card we can see that she got to this record by choosing menu option '6' (thus selecting the 6th item in the list of results). Her next action was to choose menu option 'l' which returns her to the list of returned titles.

The scrollable sequence of cards can be read left to right as a history of the entire search activity. Additional qualitative information is provided by the vertical position of a card in one of three levels; a high level activity (such as menu selections from the top part of a system's menu hierarchy) is placed on the top level, search queries are placed in the middle level and actions involving the viewing of results on the bottom level. The intention was to provide an impression of the higher level abstractions of information searching. The result is that it is possible to obtain at-a-glance impressions of certain characteristic patterns of behaviour such as the common novice 'error' (dependant upon circumstances: there are occasions when this is a sensible way of learning the language of a new subject domain) of composing a search that yields hundreds of hits and then proceeding to read through them all. Figure 2 illustrates just such a case where, using the BIDS bibliographic database, a user composed a search that yielded 4822 hits and then chose the display options and proceeded to try and read through them. The placing of the cards on the three levels is based on a small set of rules that have to be hand-coded for each database which Ariadne supports. In the 'extending the ideas' section we consider ways of tackling this limitation.

As might be expected, a search session can yield a substantial trace and so facilities are provided to elide sub-sequences of cards, replacing them by a single group card. This card and the individual cards can be annotated, either by the originator of the search or by others commenting upon it. A sequence of cards that have been elided, or folded together are indicated by a FOLD card (which is also in a different colour). The presence of an annotation is revealed, along with an indication of its size by marking the bottom section of a card. Selecting a card puts the annotation in the bottom left subwindow. Figures 1 and 2 contain examples of folds and annotations.

Studies of use

Ariadne has been used in a sequence of informal, formative evaluations during its development. We would emphasise the simplicity of the Ariadne approach. Clearly more sophisticated features could be added, but for a prototype we wanted to investigate the idea of process representation in a simple form. A key issue of concern was whether novice users would find this quite new interface and visualisation so confusing that it could not support the giving of help about the database they were using? As in the preliminary studies, volunteers undertook authentic activities, bringing along a search task that they had to undertake anyway. These were generally of a background research nature such as finding information for an essay, dissertation or literature review, or to get a sense of the literature on a field to help inform the creation or refinement of an essay topic. So far over 50 volunteers have used the system.

The users varied from teams of four Psychology undergraduates undertaking a group report-writing assignment to individual students of Women's Studies researching for a dissertation for a Master's degree. We also studied relatively expert information searchers from the Computing Department and the Library. Some of the volunteers chose to return and continue using the system for their task for up to five sessions. Sessions are recorded on either audio or video tape. We were not aiming to undertake a controlled experiment but rather an opportunistic investigation of the nature of using, learning and teaching information searching techniques and the evolving requirements of a system to facilitate this.

Given this approach, it is not appropriate for the experimenter to be passive observers. In order to maximise the degree of interaction (and also in order to offer some recompense to the volunteers for giving up their time), at the end of a session the experimenter occasionally acted as an expert browser performing an educating / facilitating role. The aim here was as usual to explore the requirements, functionality and usability of Ariadne, but this time from the perspective of a help-giver. Consequently it is inappropriate to discuss the degree to which subjects were successful in their searches. Rather we are mostly concerned with the problems that arise and how Ariadne can help or potentially help in resolving them. Clearly this can only be the first stage of an investigation, we also need to study in more depth the use by library staff.

The functionality of the system has been continually evolving and the interface has been refined during the course of the studies. Coupled with our attempt to get volunteers with as wide a range of backgrounds and abilities, this means that we can only give tentative pointers as to the effect of using the system, and mostly these concern its promise:

The recording of the interaction allows us as researchers to obtain a growing set of data about the behaviour of users when browsing databases.

Even complete novices to the use of bibliographic databases can understand the concept of the process representation. They are able to use the scroll bars to review what they or others have done and to open up a card to read its details.

When browsing, subjects (individuals and groups) forgot what they had done a few minutes previously and what they said they were going to do next. Not surprisingly they also found it difficult to remember what they had done in a previous search interaction (which may have been a week before).

Both individuals and groups could use the record to review their actions and comment on what they had done and what they were intending to do next.

Typing errors can scale up into strategic errors. The user may try a sensible strategy, make a typing error which causes the query to yield no hits, or inappropriate ones and then abandon the strategy because they are unaware of their error. The record is useful, particularly, as in the case of a very experienced librarian we observed, when the user is quite convinced that they did not make a typing error.

We observed many of the classic errors that have been noted and analysed in the literature (Tenopir, 1984). One particularly notable problem was users' lack of systematicity: they would often combine or generalise searches (a powerful browsing strategy) but fail to consider applying the technique to all their data, often because they got side-tracked by a particularly interesting finding. A related problem occurred when moving up their goalstack. Often they had a complex strategy that was composed of a number of queries. One of the query steps would lead to an interesting result. They would side-track to investigate this result, but on returning to the main strategy, fail to resume at the correct point, missing out some issues that they had intended to investigate. The record is particularly useful in such circumstances for pointing out that they have not yet exhausted all the possibilities.

Many students appear to have only a very superficial understanding of what a database is and consequently how it can and should be searched. Their mental model seems to equate the query language and search engine with an intelligent librarian. Consequently they seem to assume that the system will take account of syntactic and semantic variation. An example is a student who did not realise the searching on 'women' would fail to catch entries that contained the word 'woman'.

The record can be used as an effective teaching tool. It provides a medium for discussing the concepts and skills of information browsing. People can point to a sequence of actions and discuss what was done, whether it was effective or not, how it might have been done differently, and how the same technique could be done in another context, or even in another database. For an expert, the record is a very efficient means of determining the level of understanding of the user. Certain patterns of behaviour become dramatically evident, such as the 'get back hundreds of hits and try to read them all' error illustrated in Figure 2.

The record can be used to summarise a sequence of actions. So far this has only been observed in the case of the expert giving the novice(s) an overview of what they had done, allowing a degree of abstraction and permitting a dialogue on how to generalise the strategies that had been displayed. The scrolling and folding options are particularly useful in allowing this kind of abstraction, and were comprehensible to subjects. However, it took several iterations to achieve reasonable usability so we do not have much data of people using them effectively in an educational context. We intend to provide mechanisms to enable users to also be able to undertake this kind of reflective activity for themselves.

We found that on occasions when the experimenter was acting as a supportive expert (such as a subject librarian), it is all to easy to make mistakes about what the novices have done. In such circumstances, the record can also support an expert's faulty memory of a novice's actions. Even an expert sitting alongside a novice can forget or misinterpret what has been done. A subsequent attempt at tutoring will naturally be very unproductive in such circumstances

A main advantage of such an interface is that discussions (both remote and co-located) can then be undertaken between participants about sophisticated searching techniques without the need to learn the specialised vocabulary of the information profession. By contrast, instead of using the abstract conceptual terminology, the participants can point to, or highlight, the particular instances that were of significance in the search under discussion.

Extending the ideas

Ariadne was originally developed in C++. We have recently ported it to Java and a demonstration version (users can review existing searches but can't create new ones) is available at:

http://www.comp.lancs.ac.uk/computing/research/cseg/projects/ariadne/demo.html

We find this opportunity to demonstrate interface designs and ideas over the web an exciting prospect. We hope that it will make our ideas clearer than from just seeing two screendumps of what is crucially a dynamic system.

Our aim has been to build a tool that augments the help-giving that people undertake rather than attempting to replace it by an expert system. We hope that by providing improved versions of the ideas represented in this prototype we can improve the effectiveness and efficiency of help-giving interactions. We also believe that such an approach can be used as a complement to agent-based technologies. Inevitably there will be occasions when an agent fails to retrieve the information that a user requires. However if the attempt process of the agent can be represented and visualised, even this failure can be put to good effect by the user either considering what the agent has already tried and using their human intelligence and creativity to come up with an alternative method or by resorting to a human expert intermediary and passing on the agent's failed attempt to them as a starting point for inspiration. In this way a search process representation could meld the effectiveness of artificial and human agents (Nardi and O'Day, 1996).

We believe that a number of themes have emerged from this project that carry implications for designers of other kinds of interfaces to information retrieval systems. Firstly the mechanisms provided can be extended to support remote and asynchronous help giving. The user could record a search and then send a message requesting help to a librarian, with a pointer to her search history, suitably folded and annotated to reveal her perception of the difficulty. The librarian could add his own annotations to explain the difficulty and perhaps pass back to the enquirer a pointer to one or more search histories, either ones that the librarian has just done that solve or indicate a solution of the problem, or selections from a library of model searches that reveal (by careful annotation) a useful sequence of steps (an abstract search plan represented as a case history) that the user could reapply to their own chosen search domain. The case history library could also contain examples of classic user errors to illustrate what not to do as well as what to do.

A recent ethnographic study of activity at a library help desk (Crabtree et al, 1997) revealed in more detail a number of issues that confirm the importance of context. One example was the frequent practice of librarians turning their OPAC screen round so that the user could see it as well. This was not solely to show the results of a successful query but also at earlier stages of the help-giving interaction where the main focus of activity is for user and librarian to mutually clarify the underlying information need of the user. In order to do this, the librarian may undertake searches or choose menu options on the OPAC, and the conversation incorporates what can be seen on the screen, including participants pointing at the screen. Note that this is a kind of interaction using a computer interface that is valuable but different and supplemental to the 'conventional' understanding of the purpose of an interface.

Conventionally, an interface is viewed as a medium for communication between a user and the underlying complex functionality. In the case of an OPAC that means a way of composing queries and receiving the results, both in a form easy to learn and use by the envisaged users. In the scenario above, in addition to that conventional use, the interface is being used as a medium for discussion between two people, where the results help them establish the kind of thing that is wanted as well as (later in a help interaction) actually yielding that result. The interface is serving as a notation that can be used as a basis for clarifying dialogues that can verbally or physically point to it. Unlike notations on paper (such as schematic diagrams) that similarly can be used to supplement a dialogue, the interface-as-notation can be dynamic, changing rapidly, with the process of interactions being the focus of the dialogue as well as a single screen.

So a single-user interface is also being used collaboratively. Furthermore, this collaborative interaction is also different to the 'conventional' view of interfaces to support collaboration. The interface is not one through which the users collaborate, but acts more like a tool to enhance the existing collaboration. In this scenario, user and librarian are next to each other and so do not need computer mediated support to enable collaboration. However it is easy to envisage remote (a)synchronous help-giving contexts where the interface to the database also serves this dual purpose.

We noted in the introduction that one of the causes of a need for help giving was the rapid rate of change of systems and their interfaces. This implies a need for 'future proofing' of help support systems like Ariadne. The simplicity of Ariadne's approach means that it can be applied to any text-based interaction with a database such as can be achieved via a vt100 Telnet connection. However for each new database it will be necessary to develop a new set of rules for determining the vertical positioning of the cards in the interface. This is needed because we do not have information of the underlying semantics of the search actions - these and their interpretation are left to the human users of the system. We are currently investigating the possibilities of developing a version of Ariadne for Z39.50 compliant databases. With a Z39.50 connection, we do know the semantics of each request and so can build an interface that should be able to be used with any compliant database.

At the early stages of this project we made claims for the generality of the current text-based Ariadne which now have a rather hollow ring. We developed versions for Lancaster's OPAC and for BIDS as a proof of concept, claiming this showed it could be applied to any text based database. However both those systems now offer a web based interface. These are currently alternatives to the main text based interfaces, but we can expect increasing numbers of users to prefer the newer interface. Clearly we need to address how Ariadne can be used with a web based interface to an information system employing forms, rather than sending simple text commands. We are currently investigating this, concentrating on the development of a version of Ariadne that can sit alongside a web-based search engine such as Alta Vista. There is no essential difference between supporting a web-based search engine and a web interface to a bibliographic database. Thus we hope to recover our claims for generality in this next version.

Conclusion

We have proposed the development of interfaces for information retrieval systems that are not merely oriented to the delivery of the results of a search, but attempt to support collaboration with other users. These others may be expert intermediaries, co-workers, peer learners, etc. Such systems should help users to cope with continual and rapid change in data size, data types, systems, functionality and interfaces. One way of supporting collaboration is to provide a visualisation of the search process that can be manipulated and discussed by the users. We believe that support for collaborative use of an IR system can improve the learning and understanding both of that system and of generic search skills. In addition, by acknowledging the importance of other people in the search process, it can lead to a more robust search process, more able to cope with, and indeed exploit, the failures of whatever underlying sophisticated intelligent search mechanisms it may additionally use.

Acknowledgements

Work on the Ariadne project has been funded by the JISC New Technologies Initiative of the HEFCs and by the British Library Research and Innovation Centre. We wish to acknowledge the contributions of Gareth Smith, Jonathan Trevor and Damon Chaplin to the development of various versions of Ariadne.

References

Bates, M.J. (1989) 'The design of browsing and berrypicking techniques for the online search interface' Online Review 13, 407-24

Belkin, N.J. and Marchetti, P.G. (1990) 'Determining the functionality and features of a intelligent interface to an information retrieval system' Proceedings of the the 13th International Conference on Research and Development in Information Retrieval (SIGIR'90)

Borgman, C.L. (1996) 'Why are online catalogs still hard to use?' Journal of the American Society for Information Science 47, 493-503

Collins, A. and Brown, J.S. (1988) 'The computer as a tool for learning through reflection' in Mandl, H. and Lesgold, A. (ed) Learning Issues for Intelligent Tutoring Systems, Springer-Verlag

Crabtree, A., Twidale, M.B., O'Brien, J. and Nichols, D.M. (1997) 'Talking in the Library: Implications for the Design of Digital Libraries' Technical Report CSEG/1/97 Computing Department, Lancaster University

Foss, C.L. (1987) 'Learning from errors in Algebraland' Technical Report IRL-87-0003 Institute for Research on Learning, Palo Alto, California

Fowell, S. and Levy, P. (1995) 'Developing a new professional practice: a model for networked learner support in higher education' Journal of Documentation 51, 271-80

Grosser, K. (1991) 'Human networks in organizational information processing' Annual Review of Information Science and Technology 26, 349-402

Grudin, J. (1994) 'Groupware and social dynamics: eight challenges for developers' Communications of the ACM 37, 92-105

Hollan, J.D., Hutchins, E.L. and Weitzman, L. (1984) 'STEAMER: an inspectable simulation-based training system' AI Magazine 5, 15-27

Hoppe, H.U. and Zhao, J. (1994) 'C-TORI: an interface for cooperative database retrieval' in Karagiannis, D. (ed) Database and Expert Systems Applications, Springer-Verlag

Katz, S. and Lesgold, A. (1994) 'Implementing post-problem reflection within coached practice environments' Proceedings of the East-West International Conference on Computer Technologies in Education (EW-ED'94) Vol. 1 125-30

Lee, A. (1992) Investigations into History Tools for User Support PhD Thesis, Department of Computer Science, University of Toronto

Lemaire, B. and Moore, J. (1994) 'An improved interface for tutorial dialogues: browsing a visual dialogue history' Proceedings of the Conference on Human Factors in Computing Systems (CHI'94) 16-22

Levy, D.M. and Marshall, C.C. (1995) 'Going digital: a look at assumptions underlying digital libraries' Communications of the ACM 38, 77-84

Mackinlay, J.D., Rao, R. and Card, S.K. (1995) 'An organic user interface for searching citation links' Proceedings of the Conference on Human Factors in Computing Systems (CHI'95) 67-73

Nardi, B.A. and Miller, J.R. (1991) 'Twinkling lights and nested loops: distributed problem solving and spreadsheet development' International Journal of Man Machine Studies 34, 161-84

Nardi, B.A. and O'Day, V. (1996) 'Intelligent agents: what we learned at the library' Libri 46, 59-88

Sugimoto, S., Gotou, S., Zhao, Y., Sakaguchi, T. and Tabata, K. (1995) 'Enhancing usability of network-based library information system - experimental studies of a user interface for OPAC and of a collaboration tool for library services' Proceedings of Digital Libraries '95

Tenopir, C. (1984) 'To err is human: seven common searching mistakes' Library Journal 109, 635-36

Twidale, M.B., Nichols, D.M. and Paice, C.D. (to appear) 'Browsing is a collaborative process' Information Processing & Management

Twidale, M.B., Nichols, D.M., Smith, G. and Trevor, J. (1995) 'Supporting collaborative learning during information searching' Proceedings of Computer Support for Collaborative Learning (CSCL 95) 367-74

ARIADNE | CSEG | Computing Department | Lancaster University