David Madigan[124], C. Richard Chapman[12], Jonathan Gavrin[12], Ole Villumsen[13], John Boose[2]

[1]University of Washington, [2]Fred Hutchinson Cancer Research Center, [3]Aarhus University

[4]Department of Statistics, GN-22

University of Washington

Seattle, WA 98195

fax : (206) 685-7419

e-mail: {madigan, crc, jgavrin}@u.washington.edu, ovillumsen@daimi.aau.dk, jboose@spock.fhcrc.org

ABSTRACT

Creation and maintenance of links in large hypermedia documents is difficult. Motivated by an application to a federal clinical practice guideline for cancer pain management, we have developed and evaluated a repertory grid-based linking scheme we call repertory hypergrids. Harnessing established knowledge acquisition techniques, the repertory hypergrid assigns each "knowledge chunk" a location in "context space". A chunk links to another chunk if they are both close in context space.

To evaluate the scheme, we conducted a protocol analysis. Six users of the guideline addressing typical cancer pain management tasks made 30 explicit links. The repertory hypergrid using a neighborhood size of 16 captures 24 of these links. With optimization, the repertory hypergrid captures 27 of the links with a neighborhood size of 13.

KEYWORDS: Implicit linking, repertory grid, clinical practice guidelines, link maintenance, evaluation.

1 INTRODUCTION

We are developing Talaria[3], a hypermedia training and reference tool for healthcare providers managing patients with cancer pain. The clinical practice guideline on cancer pain relief, just released by the Agency for Health Care Policy and Research (AHCPR), formally defines much of the knowledge base for Talaria [22]. The purpose of the program mirrors that of the practice guideline: to improve the management of pain in patients with cancer by informing physicians, nurses and other health care providers about current therapeutic options and principles. Talaria addresses many of the problems associated with booklet-based clinical practice guidelines.

We have developed a novel hypermedia linking scheme to meet Talaria's requirements. The scheme implicitly constructs links between "knowledge chunks" by assigning each chunk a location in a "context space". A chunk links to another chunk if they are close in context space. To evaluate the effectiveness of the approach we conducted a protocol analysis. The results suggest that the linking scheme is effective and overcomes many of the difficulties associated with large hyper-linked documents. We begin with a sketch of the problem domain.

1.1 Cancer Pain and the AHCPR Guideline

Pain is a pernicious force that increasingly threatens the functional capability and psychologic well-being of the cancer patient as disease progresses. Because of unrelieved pain, many patients spend the last weeks, months or even years of their lives with needless discomfort and disability [8]. Tragically, the extensive suffering caused by cancer pain is largely unnecessary. Gains in knowledge about pain and its control and technological advances in pain management now enable informed physicians to relieve up to 90% of cancer pain [25,34]. However, many patients get inadequate relief because of underuse of treatment resources. This largely stems from a lack of knowledge amongst caregivers. Talaria, like the AHCPR guideline on cancer pain, addresses this obstacle.

The AHCPR defines clinical practice guidelines as "systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances". The AHCPR sponsors private sector panels, composed of experts from relevant disciplines, to develop these guidelines to reduce clinical uncertainty and improve patient outcomes. The cancer pain guideline is a 260-page, paperback booklet consisting of text, tables, and figures. It addresses pain assessment, the psychological and physiological impact of cancer pain, interventions for the treatment of cancer pain including pharmacological, psychosocial and procedurally based interventions, and a variety of special topics. The target audience for the guideline extends to patients (both adults and children), patients' families and clinicians at all levels.

Clinical practice guidelines of non-federal origin have proliferated in recent years. The American Medical Association alone offers 1,300 different guidelines. The AHCPR estimates that over 10,000 guidelines have been developed in the history of medical practice. While the cancer pain guideline is our immediate focus, we intend to develop a general purpose methodology.

1.2 Talaria Objective and Requirements

Much anecdotal evidence exists that guidelines impact practice patterns minimally [15]. More formal studies such as those of Grilli et al. [16], Lomas et al. [26], and Ford et al. [13] report similar findings. Our objective is to render guidelines in a more useful form. We have observed many deficiencies in the paper-based guidelines and these largely define the requirements for Talaria [12,28]:

* Booklet guidelines have little depth and provide no support for users in specific specialties who want explanations of concepts, mechanisms, and procedures.
* Booklet guidelines cannot demonstrate procedurally based interventions.
* Booklet guidelines lack the facility to implement an instructional strategy and provide no feedback to the reader.
* Booklet guidelines provide minimal support for practical day-to-day problems such as drug dose calculations, or dose conversions across routes of delivery.
* While guidelines in booklet form facilitate distribution, such booklets typically get absorbed into the mound of literature accumulated by every healthcare provider.
* Booklet guidelines provide little motivation for clinicians to learn new information.
* Booklet guidelines provide little support for patient-clinician communication.

Booklet guidelines do, however, provide an effective browsing tool and a key requirement is to emulate this ability.

Hypermedia can support browsing as well as address the above deficiencies. However, in large hypermedia documents, manual creation and maintenance of links prove difficult [3,9,37,27,38,40]. Each time the document acquires a new knowledge chunk, the author must examine all the existing chunks to determine what new links are required. Tuning links to optimize performance presents similar difficulties. The cancer pain guideline has 136 sections, 18 Tables, and a variety of supporting material. Furthermore, the guideline is evolving: the AHCPR plans to issue new editions every two or three years. We require a linking scheme that does not demand hand-creation and maintenance of each link.

1.3 Overview of the Paper

In Section 2 we describe the linking scheme we have developed. In Section 3 we present an evaluation of aspects of the scheme. In the final section we describe our current activities and future directions.

2 LINKING SCHEME

While a traditional document is a collection of text organized into chapters, sections, and paragraphs, and arranged in a linear fashion, a hypermedia document is a collection of knowledge chunks arranged in a network. A knowledge chunk can be a piece of text, a movie, an animation sequence, a picture, an audio clip, a database or even a calculating tool (see, for example, Anderson [1], or Sheplock et al. [36]). Links connect knowledge chunks that have a semantic connection: if one chunk triggers an association with another, then the author links them and the user can get from one chunk to the other.

We refer the reader to Nielsen [32] and Balasubramanian [2] for brief histories of hypertext and descriptions of the best known applications. Note however, there have been only a few reports on efforts to evaluate the effectiveness of such applications formally; evaluation is central to our approach.

2.1 An Implicit Linking Scheme

Kibby and Mayes [24] suggested a linking scheme that obviates the need for explicit links (although it does not preclude explicit links). Our work has focused on extending their ideas, linking them to recent progress in knowledge acquisition, and evaluating their effectiveness. The fundamental idea is to independently assign each chunk a location in a high-dimensional "context space". Chunks that are close together form a "neighborhood" and link implicitly since they share a similar context. The author or the user defines the neighborhood as the nodes within a certain distance of the current node, or as the nearest "n" nodes. We currently adopt the latter approach in Talaria and typically choose n to be between 10 and 30 to yield a manageable neighborhood size.

The decisive advantage of this scheme is modularity: an author can link a new or modified knowledge chunk to its associated chunks simply by rating it against each of a number of "traits", thereby eliminating the requirement to examine all existing chunks.

2.2 Repertory Hypergrids

We define the context of each knowledge chunk with a high-dimensional trait vector. Each element of a trait vector is a number representing the strength of association between the chunk and the corresponding trait. For example, in the cancer pain guideline, Section 3.3.2 (Dosage Titration) rated a 6 on "Drug" and a 2 on "Pain Assessment" on a scale from 1 to 6. We score every knowledge chunk against every trait. Knowledge chunks that have similar ratings on a large number of traits will be relatively near each other in the space spanned by the traits and will thus be implicitly linked. Knowledge chunks that have rather different trait vectors will be far apart in this context space, and will not link to each other.

We show in Table 1 a sample of trait vectors and corresponding traits for six chunks of the AHCPR Cancer Pain Guideline (here we equate knowledge chunks with sections, tables, and figures in the guideline). Boose [5] referred to such a table as a "repertory grid". In this context we call it a "repertory hypergrid". We believe that researchers have not previously explored repertory grids in the hypermedia context.

                                              Sections                 
             Traits               2.3.1  2.3.2  T4    2.3.3  2.3.4  2.3.5  
        Pain Assessment           2      2     1      1      1      1     
  Barriers to Pain Management     1      1     1      1      1      1     
              Bone                6      5     5      2      2      1     
     Central Nervous System       3      6     6      4      1      1     
              Skin                1      1     1      1      4      5     

Table 1: A repertory grid with five traits and six sections of the draft AHCPR cancer pain guideline. Here we used a 6-level rating scale.

Boose et al. [7] discussed the advantages and disadvantages of various nominal, ordinal, and continuous rating scales. We have used a six-level ordinal rating scale for Talaria, as shown in Table 2. We return to this issue in Section 4 below.

 Rating    Operational Definition     
    6      Chunk is precisely         
           concerned with this trait  
    5      Trait is a secondary       
           topic in the chunk         
    4      More than a passing        
           reference; less than a     
           secondary topic            
    3      Explicit passing           
           reference                  
    2      Implicit passing           
           reference                  
    1      No mention implicit or     
           otherwise                  

In the next three subsections we discuss the construction of repertory hypergrids.

2.2.1 Triadic Elicitation of Traits

From where do the traits come? Waltz and Pollack [41] suggest that traits "should be chosen on the basis of first principles to correspond to the major distinctions humans make about situations in the world." We have found this rather general advice difficult to implement in practice. Fortunately, Boose and his colleagues [4,5,7] provided a formal methodology and software tools (MacQuinas and Dart) for identifying and analyzing traits. They based their approach on Kelly's personal construct theory [23].

In MacQuinas, an "expert" first lists the possible solutions to a problem such as a medical diagnosis (the solutions in that case would be diagnostic categories). These correspond to our knowledge chunks. Next, the expert specifies a collection of traits as follows: MacQuinas presents the solutions three at a time and asks the expert to identify what feature best distinguishes any one solution from the others. Kelly suggested these triads for efficiently identifying minimal sets of traits. Once the expert has identified the traits, he/she rates each solution against each trait to create the repertory grid.

2.2.2 Grid Analysis Tools

MacQuinas contains a wealth of tools for analyzing repertory grids, including trait implication graphs, trait and knowledge chunk cluster analyses, principal components analysis, hierarchical organization of traits and knowledge chunks, "laddering tools" for expanding and contracting traits, and tools for combining the trait-spaces of multiple experts. Boose et al. [7] describe many of these techniques for analyzing repertory grids. We have found cluster analysis particularly useful for identifying poorly discriminated chunks and redundant traits.

We also use multidimensional scaling (MDS) to provide an approximate three-dimensional projection of the knowledge chunks. We then use brushing and spinning tools to visualize the chunks in 3-D. In Figure 1 we show a two-dimensional projection of 20 of the sections in the cancer pain guideline. This shows the relative location of the 20 knowledge chunks in a two-dimensional projection of context space.

Figure 1: MDS 2-Dimensional view of context space. This shows 18 sections from the AHCPR Cancer Pain Guideline. We note the similarity with the spatialized text plots of Marshall and Shipman [30].

2.3 Interface Design and the Travel Metaphor

Talaria's interface design derives principally from the concept of a Learning Support Environment (LSE) developed and implemented by Hammond and co-workers [18,19]. A similar approach characterizes the Hyperties system of Morariu and Schneiderman [31] and Marchionini and Schneiderman [29]. An LSE provides the learner with a set of tools to support exploration of, or instruction in, some field of knowledge. The tools include both aids for accessing information and for learning [28]. We use a travel metaphor to structure the navigation tools and provide the user with an intuitive context mechanism. Each screen display, be it text, animation sequence, image or movie, represents a place to visit. The various access facilities represent the ways and means of traveling around. Two explicit forms of navigation through the materials reflect the extremes of learner-controlled and system-controlled access. These are go-it-alone travel and guided tours [39].

As Halasz [17] indicated, navigational tools alone are not sufficient. The metaphor encompasses a range of access facilities along with the navigation tools: maps give users "bird's eye" views of the knowledge available and they can zoom in on any chosen chunk. An index provides a mechanism for keyword-based access. Ultimately both the maps and the index in Talaria could adapt to individual users' needs. Ichimura and Matsushita [21] suggest that maps may be problematic: it is hard to generate maps that communicate effectively the contents of the nodes. A combination of fisheye views (see, for example, [33]) and pop-up chunk summaries may alleviate this problem. We are currently implementing these in Talaria.

3 IMPLEMENTING THE SCHEME FOR THE CANCER PAIN GUIDELINE

Using a single physician expert (JG), we have constructed a repertory hypergrid for the complete AHCPR guideline with 29 traits and 136 knowledge chunks[4]. The sections and subsections of the guideline define the knowledge chunks.

3.1 Traits

Triadic elicitation of traits proceeded as follows: First, we selected 20 sections from the guideline. We chose these to be representative of the material in the guideline. Next, we randomly selected sets of three from this set of 20 sections. For each triad, a single expert (again JG) identified a trait which "best distinguished one of these from the other two." He endeavored to choose traits relevant for navigation. We continued in this fashion until five consecutive triads failed to elicit a new trait. At this point we had 29 distinct traits. We show these in Table 3.

Surgical                                             
Respiratory                
Psychololgic                                       
Sedative                   
Education                                            
Ethics                     
Non-Pharmacologic Management          
Diagnosis                  
Procedural Pain                                  
Sleep                      
Analgesics                                          
Opioid Analgesic           
Adverse Outcome                                
Non-Opioid Analgesic       
Pain Assessment                                 
Family                     
Barriers to Pain Management              
Regulations                
Bone                                                   
Demographics               
Central Nervous System                     
Anesthesia                 
Skin                                                    
Indications                
Treatment Modalities                          
Mechanisms of Pain         
Monitoring Patients                             
Mechanisms of Pain Relief  
Drug                                                  

Table 3: The 29 traits used for the AHCPR Cancer Pain Guideline

We note that some traits are specializations of others. For example, "Opioid Analgesic" is a special case of "Analgesics". Currently we treat all traits on an equal footing. We are investigating alternative approaches.

3.2 Rating Procedure and Grid Analysis

The rating of the 136 chunks against the 29 traits required approximately 80 man-hours. To ensure consistency and accuracy, it is essential to have at least two people participating--the expert and a knowledge engineer. Careful rating proved worthwhile: we re-scored a randomly selected 20 sections more than one month after the initial rating and never differed by more than one on the six-point scale. The key to such consistency is a clearly defined rating scale (Table 2) and unambiguous trait descriptions. The complete grid is available from the first author.

Grid analysis addresses two questions: do we need all the traits we have? Do we have all the traits we need? We can address the former question with an analysis of the grid as discussed in Section 2.2.2 above and Section 3.3.3 below. However, the latter question requires dynamic analysis: do the links defined by the grid correspond to the links users make when using the guideline to address cancer pain management tasks? We now describe such an analysis.

3.3 Evaluation Methodology

The key output of the repertory hypergrid is a matrix of the pairwise distances between all the knowledge chunks. The linkplots of Bernstein et al. [3] provide an insightful method for viewing the links implied by this matrix for different neighborhood sizes; see Figure 2.

Figure 2: Linkplots for the 136 chunks in Talaria. Each dot represents a link. The plot on the left uses a neighborhood size of 16 chunks while the plot on the right uses 30 chunks. The chunks are numbered in the order in which they appear in the cancer pain guideline. The rectangular structures in the plots reveal the chapter structure of the book. Note the linking scheme makes many links between chunks in different chapters in the guideline.

In earlier pilot work, we asked several potential users of Talaria to assess a sample of such pairwise distances and then we compared those distances with those suggested by the repertory hypergrid. The correspondence was close although the task seemed contrived. Ultimately we intend to compare competing repertory hypergrids with various distance metrics and trait and node weighting schemes on the basis of user performance in locating information in the hypermedia tool. Nielsen [32] described several hypermedia usability tests. Initially however, rather than confound the linkage analysis with software-specific issues such as interface design, we conducted a protocol analysis (see, for example, [6], and the references therein) using the guideline booklet itself.

We selected four cancer pain management tasks from the case studies prepared by the Wisconsin State Cancer Pain Initiative. These case studies are prototypical of cancer pain management problems; instructional workshops throughout the country use them.

We selected six subjects to participate in the analysis: a senior pain service physician (A), a primary care internist (B), a senior pain service nurse (C), a pain service resident physician (D), a family practitioner (E), and a pediatric nurse practitioner (F). We instructed the subjects to use the guideline booklet to address the four tasks and to "describe as fully as possible your thoughts as you browse the guidelines for the information you seek to address each task." We recorded the subjects' use of the book throughout and subsequently used the video recording for clarification. The sessions ranged from 45 to 90 minutes.

When a subject successively visited two useful sections while addressing a single task, we deemed that to be a link. The subjects visited 24 distinct chunks and made a total of 30 links. In the future we intend to add more open-ended tasks to explore other chunks in the book.

3.3.1 Evaluation of Linking Scheme

For each of the links made by the subjects, we calculated the Euclidean distance from the source of the link to all the chunks in the document (we discuss other distance metrics below). We then calculated the rank order of the destination of the link. For example, a subject made a link from Section 7.4 (Substance Abusers) to Section 1.5.1 (Legal Regulation of Opioids). We rank ordered the distances from Section 7.4 to all the other chunks and found that Section 1.5.1 was the fifth nearest. Thus we would assign the link from 7.4 to 1.5.1 a rank order of five.

Clearly we would like the destination of ecah link to be in the neighborhood of its source. Equally, we seek to minimize cognitive overload, and therefore want the neighborhoods to be small [11]. Taken together, we require the rank order of the links made by the subjects to be as small as possible. We show a plot of the rank orders of the 30 links in Figure 3.

Figure 3: Plot of the percentage of the links made by the users in the protocol analysis against neighborhood size. Ideally, small neighborhoods would capture most or all of the links made by the subjects. A neighborhood of size n includes the n nearest chunks.

The figure shows that Talaria requires a neighborhood size of 30 to capture all the links made by the subjects in the protocol analysis. A neighborhood size of 16 captures 24 of the 30 links. Thus we have shown that for a range of tasks and users, the repertory hypergrid, with a manageable neighborhood size, captures the links made by the subjects.

Figure 4: Boxplots of the rank orders of the implicit links formed from the repertory grid analysis using seven different distance metrics. The boxplots show the minimum, the 25th percentile, the median, the 75th percentile, and the maximum excluding outliers.

3.3.2 Distance Metric Evaluation

While Euclidean distance represents what is, perhaps, the most natural distance metric, we carried out a detailed investigation of six different metrics in addition to Euclidean distance. We show the results in Figure 4.

When comparing two chunks' trait vectors, we define the distance metrics as follows: "city" block distance is the sum of the absolute pairwise differences; "Rsquare" is the Pearson correlation co-efficient squared; "1/>1 bin" is the sum of the pairwise differences having dichotomized the six point scale into "1" and "bigger than 1"; "<6/6 bin" is the same thing but dichotomizing into "less than 6" and "6"; "Minimum" is sum of the pairwise minima; "Weighted" is the sum of six minus the absolute pairwise differences weighted by the pairwise minima.

From Figure 4, we see that Euclidean distance, city block distance, and RSquare provide the best overall performance. Rsquare has the lowest median, but Euclidean has smaller overall variation and upper percentiles.

3.3.3 Trait Deletion

Deleting five of the traits: Adverse outcome, Pain assessment, Central nervous system, Skin, and Demographics improves performance: using the Euclidean distance metric, a neighborhood size of 18 now captures all 30 of the links made by the subjects and a neighborhood size of 14 captures 27 of the links. However, before removing these traits permanently, we intend to assess their utility in addressing a broader range of tasks.

4 DISCUSSION

We have proposed an implicit linking scheme and used it to develop a hypermedia implementation of the AHCPR cancer pain guideline. Our protocol analysis suggests that the scheme captures efficiently the links made by users of the guideline. Clinical practice guidelines must be current; hypermedia has much to offer here.

We currently are exploring many extensions to the basic scheme:

* The scheme does not preclude having a small number of additional author- or user-specified explicit links.

* The inter-chunk distance provides a mechanism for implementing a scaled rather than binary link. Salton et al. [35] make a similar proposal, but in the context of text retrieval.

* Chang's HieNet by default creates a link between the smallest pair of nodes in a neighborhood [10]. This notion of parsimony may be useful in the guideline context.

* Currently the links in Talaria are untyped and connect entire chunks: we may wish to explore generalizing this in the future.

We adopted a six point rating scale in Talaria. Kibby and Mayes [24] suggest a binary scheme and it is easy for authors to specify. They base their approach on the human memory models of Hintzman [20], who uses a three-point scale. Waltz and Pollack [41], and later Gallant [14], present an essentially identical approach, but in the context of natural language recognition. They adopted 4-point and 5-point scales respectively. Boose et al. [7], in yet another context, suggested different scales for different traits, and Anderson [1] uses a 7-point scale. Initially we used a binary rating scale but our analysis suggests that a finer rating scale provides superior performance (see Figure 4).

ACKNOWLEDGEMENTS

The authors are grateful to Jeff Bradshaw, Peter Dunbar, and Robert Jacobsen for helpful contributions. This research is funded in part by a NIH SBIR grant to Statistical Sciences Inc., NCI grant CA 38552, and the Danish Research Academy.

REFERENCES

[1] Anderson, N. "Medical center: a modular hypermedia approach to program design", In: Sociomedia: Multimedia, Hypermedia, and the Social Construction of Knowledge, E. Barrett (Ed.), MIT Press, Cambridge, pp. 369-389, 1992.

[2] Balasubramanian, V. Hypermedia Issues and Applications, A State-of-the-Art Review, Independent Research Report as part of Ph.D. Program, Graduate School of Management, Rutgers University, December 1993.

[3] Bernstein, M., Bolter, J.D., Joyce, M., and Mylonas, E. "Architectures for volatile hypertext", Hypertext 91: Proceedings of the Third ACM Conference on Hypertext, ACM Press, pp. 243-260, 1991

[4] Boose, J.H. "A knowledge acquisition program for expert systems based on personal construct psychology", International Journal of Man-Machine Studies 23, pp. 495-525, 1985.

[5] Boose, J.H. Expertise Transfer for Expert System Design. Elsevier, New York, 1986.

[6] Boose, J.H. "Knowledge acquisition tools, methods, and mediating representations" H. Motoda, R. Mizoguchi, J. Boose, B. Gaines (Eds.), Knowledge Acquisition for Knowledge-Based Systems, IOS Press, Ohmsha, Ltd., Tokyo, 1990.

[7] Boose, J.H., Shema, D.B., and Bradshaw, J.M. "Recent progress in AQUINAS: a knowledge acquisition workbench", Knowledge Acquisition, 1, pp. 185-214, 1989.

[8] Bonica, J.J. "Cancer pain", The Management of Pain. Lea and Febiger, Malvem, PA, second edition, 1990.

[9] Canter, D., Rivers, R., and Storrs, G. "Characterising user navigation through complex data structures", Behaviours and Information Technology, 4, pp. 93-102, 1985.

[10] Chang, D.T. "HieNet: A user-centered approach for automatic link generation", Hypertext 93: Proceedings of the Fifth ACM Conference on Hypertext, Seattle, pp. 145-158, 1993.

[11] Conklin, J. "Hypertext: A survey and an introduction", IEEE Computer, 20, pp. 17-41, 1987.

[12] Cook, P. "An encyclopedia publisher's perspective, Interactive Multimedia", Apple Computer, Inc., Microsoft Press, 1988.

[13] Ford, L., Hunter, C., Diehr, P., Frelick, R., and Yates, J. "Effects of patient management guidelines on physician practice guidelines: the Community Hospital Oncology Program experience", Journal of Clinical Pharmacology, 5, pp. 504-511, 1987.

[14] Gallant, S.I. "A practical approach for representing context and for performing word sense disambiguation using neural networks", Neural Computation, 3, pp. 293-309, 1991.

[15] Gardner, E. "Putting guidelines into practice", Modern Healthcare, pp. 24-36, 1992.

[16] Grilli, R., Alexanian, A., Apolone, G., Marsoin, S., Nicolucci, A., Torri, V., Compagnucci, M., and DiMambro, E. et al. "The impact of cancer treatment guidelines on actual practice in Italian general hospitals: The case of ovarian cancer", Annals of Oncology, 1, pp. 112-118, 1991.

[17] Halasz, F. "Reflections on NoteCards: Seven issues for the next generation of hypermedia systems", Communications of the ACM, 31, pp. 836-852, 1988.

[18] Hammond, N.V. "Hypermedia and learning: who guides whom?" Computer assisted learning--ICCAL 89, H. Maurer (Ed.), Springer-Verlag, pp. 167-181, 1989.

[19] Hammond, N.V. and Allinson, L.J. "Travels around a learning support environment: rambling, orienteering or tutoring?", CHI '88 Conference Proceedings: Human Factors in Computer Systems, E. Soloway, D. Frye, and E.B. Sheppard (Eds), ACM Press, pp. 269-273, 1988.

[20] Hintzman, D.L. "Schema Abstraction in a multiple-trace memory model", Psych Rev, 93, 4, pp. 411-428, 1986.

[21] Ichimura, S. and Matsushita, Y. "Another dimension to hypermedia access", Hypertext `93: Proceedings of the Fifth ACM Conference on Hypertext, Seattle, pp. 63-72, 1993.

[22] Jacox, A., Carr, D.B., Payne, R., et al. Management of Cancer Pain. Clinical Practice Guideline No. 9. AHCPR Publication No. 94-0592. Rockville, MD. Agency for Health Care Policy and Research, U.S. Department of Health and Human Services, Public Health Service, March 1994.

[23] Kelly, G.A. The Psychology of Personal Constructs. New York: Norton, 1955.

[24] Kibby, M.R. and Mayes, J.T. "Towards intelligent hypertext", Hypertext: theory into practice, R. McAleese (Ed.), Ablex, New Jersey, pp. 164-172, 1989.

[25] Levy, M.H. "Pain management in advanced cancer", Semin Oncol., 12, pp. 394-410, 1985.

[26] Lomas, J., Anderson, G., Dominick-Pierre, K., Vayda, E., Enkin, M., and Hannah, W. "Do practice guidelines guide practice: The effect of a consensus statement on the practice of physicians", New England Journal of Medicine, 321, pp. 1306-1311, 1989.

[27] Lucarella, D., Parisotto, S., and Zanzi, A. "MORE: Multimedia object retrieval environment", Hypertext `93: Proceedings of the Fifth ACM Conference on Hypertext, Seattle, pp. 39-50, 1993.

[28] Madigan, D. and Chapman, C.R. "Multimedia tools for cancer pain education", Proceedings of the First ACM Workshop on Multimedia in Medical Education, H. Ng, J. Ong, and R. Rada (Eds.), In press.

[29] Marchionini, G. and Schneiderman, B. "Finding facts and browsing knowledge in hypertext systems", IEEE Computer, 21, 1988.

[30] Marshall, C.C. and Shipman, F.M. "Searching for the missing link: Discovering implict structure in spatial hypertext", Hypertext `93: Proceedings of the Fifth ACM Conference on Hypertext, Seattle, pp. 217-230, 1993.

[31] Morariu, J. and Schneiderman, B. "Design and research on the interactive encyclopedia system (TIES)" Proceedings of the 29th Conference of the Association for the Development of Computer-based Instructional Systems, pp. 19-21, 1986.

[32] Nielsen, J. Hypertext and Hypermedia. New York: Academic Press, 1990.

[33] Noik, E.G. "Exploring large hyperdocuments: Fisheye views of nested networks", Hypertext `93: Proceedings of the Fifth ACM Conference on Hypertext, Seattle, pp. 192-199, 1993.

[34] Portenoy, R.K. "Cancer pain: epidemiology and syndromes", Cancer. 63, pp. 2298-2307, 1989.

[35] Salton, G., Allan, J., and Buckley, C. "Automatic structuring and retrieval of large text files", Communications of the ACM, 37, pp. 97-108, 1994.

[36] Sheplock, G.J., Thomas, P.S., and Camporesi, E.M. "An interactive computer program for teaching regional anesthesia", Anesthesiology Review, 20, pp. 53-59, 1993.

[37] Tang, H., Bardn, R., and Clifton, C. "A new learning environment based on hypertext and its techniques", Advanced Research on Computers in Education, R. Lewis and S. Otsuki (Eds.), North-Holland, pp. 119-127, 1991.

[38] Tompa, F.W., Blake, G.E., and Raymond, D.R. "Hypertext by link-resolving components", Hypertext `93: Proceedings of the Fifth ACM Conference on Hypertext, Seattle, pp. 118-130, 1993.

[39] Trigg, R. "Guided tours and tabletops: Tools for communicating in a hypertext environment", ACM Transactions on Office Information Systems, 6, pp. 398-414, 1988.

[40] Utting, K. and Yankelovich, N. "Context and orientation in hypermedia networks", ACM Transactions on Information Systems, 7, pp. 58-84, 1989.

[41] Waltz, D.L. and Pollack, J.B."Massively parallel parsing: a strongly interactive model of natural language interpretation", Cognitive Science, 9, pp. 51-74, 1985.