
Summary of curl v2.0
====================

Andrew Davison
The Department of Computer Science
The University of Melbourne
Parkville, Victoria 3052, Australia
Email: ad@cs.mu.oz.au
Tel: +61 3 9287 9172
Fax: +61 3 9348 1184
URL: http://www.cs.mu.oz.au/~ad

May 1996


1. Introduction
===============

This is a brief summary of curl version 2.0, which has been extracted
from the user guide, guide.ps, which is in the 'docs' directory of the
curl release. The text-only version of the user guide is also there,
in guide.txt.

curl is a tool for automatically generating the links between the pages
comprising a World Wide Web (WWW) document. curl examines a contents
file containing the file names of the individual pages, and modifies
each file to include the necessary links to its neighbours, parent,
initial page, and to pages listing the figures, tables, keywords and
new pages in the document. These figures, tables, keywords, and new
additions pages are generated automatically by curl. curl can also
produce two kinds of contents lists, which can be automatically
inserted into pages.

There is a search engine, available from every page, which can scan the
document for specified strings.  The matching pages can be accessed
from a search results page via dynamically generated links.

If the author wishes to add extra pages, remove some, or to reorder
pages, this is achieved by modifying the contents file. When curl is
invoked on the new version of the contents file it will modify the
links necessary to reflect the new WWW document structure.

curl comes with a range of additional tools. For instance the frequency
of word occurrences in the document can be analysed in a number of
ways. This is useful when the document hierarchy is being developed, or
for deciding on keywords.

A set of tools are included for linking pages that cannot be connected
easily within a document hierarchy.  An example would be a page in the
humour section of a newsletter and an article in the editorial section
which both discuss street maps. They already have well-defined
positions in the document (in the humour and editorial sections) but
could also be usefully inter-linked to capture the street maps theme.

The next section motivates curl by means of a small example which is
used in later sections of the user guide. The code for the example is
in the 'example' directory of the curl release. Section 3 presents an
overview of curl.

A medium sized document (consisting of over 350 pages) built using
curl can be found at http://www.cs.mu.oz.au/ad/alp/archive.html.



2. Motivation
=============

The motivation for this work was the realisation that large WWW
documents, consisting of tens (perhaps 100's) of pages linked together,
are incredibly difficult to build, maintain, change, and extend. This
is *not* due to the HTML coding of the information on the pages, but
rather because of the numerous links *between* the pages.

For instance, a common kind of WWW document configuration is shown
below:


                          newsletter
                              |
                              |
            --------------------------------------
            |                 |                  |
         editorial          humour          related-info
            |                 |
            |                 |
            |            -----------
            |            |         |
            |         streets     nono
            |
            |
    -------------------
    |       |         |
  marks  pathology  whome


The document is hierarchical, hence its tree-like form, with each node
corresponding to a page of the document. 'newsletter' introduces the
newsletter, and contains links to the 'editorial' and 'humour'
sections, and to the 'related-info' page.  The 'editorial' and 'humour'
sections consist of a series of related pages which are reached by
links.

The tree format is very common: its hierarchical nature reflects the
chapter, section, sub-section partitioning used in books, magazines,
manuals, etc. However, representing this as a WWW network requires an
enormous amount of work, because of the large number of links between
the pages needed to support browsing over related topics, and movement
between sections. For instance, 'pathology' should have links to
'marks', 'whome' (related pages), 'editorial' (the section header), and
'newsletter' (the overall contents page). 'editorial' should have links
to 'marks', 'pathology', 'whome' (its articles), 'humour' and
'related-info' (other sections), and 'newsletter' (the contents page).

Building these links is error-prone; changing them is probably even
harder to get right. For example, if 'pathology' was moved after
'whome', then some or all of the links in 'marks', 'pathology', 'whome'
and 'editorial' would need to be changed.

Documents of this kind often have separate pages listing the tables,
figures and keywords used. These pages increase the complexity of the
links in the document even further.

As the size of a document increases, it becomes harder to find
things even when the document has a keyword index and a
hierarchical structure. Therefore a search engine is essential for
scanning the document and dynamically generating a page of links to
the interesting pages. This links page can be viewed as a simple kind
of 'personalised tour', based around the user's interests.

Often documents cannot be structured in a completely hierarchical
manner. For instance in the example above, 'whome' and 'streets' may
share a common theme which should be represented by a link between
them. curl comes with tools for generating such links, while retaining
the overall document hierarchy.



3. An Overview of curl Features
===============================

There are 10 main features:

	1. Automatic link generation.
	2. Support for incremental link change.
	3. Hierarchical structuring.
	4. Recording of keywords, figures and tables.
	5. Link style files.
        6. A search engine.
        7. A new additions page.
        8. Easy removal of curl-generated links.
        9. Tools for analysing word occurrences.
       10. Tools for manipulating links outside the document hierarchy.

These features are described in the following sections.


3.1. Automatic Link Generation
------------------------------
curl addresses the linkage problem by automatically generating all the
links between the pages of a hierarchical WWW document. All the author
needs to do is write a contents file. For the example in section 2,
this would be:

   newsletter {
     editorial {
       marks
       pathology
       whome
     }
     humour {
       streets
       nono
     }
     related-info
   }

Each name corresponds to the HTML file which holds the document page
(e.g. the 'marks' page is stored in 'marks.html').  Although the pages
are assumed to be correctly written in HTML, they do not need to
include any links which reflect the document format defined by the
contents file (there may be links, but these will be for other
purposes).

Using the contents file, curl will modify the HTML files to include all
the necessary links to create the document. It can also generate two
kinds of contents listings.


3.2. Support for Incremental Link Change
----------------------------------------
Changing links is simply a matter of reordering the contents file and
reexecuting curl. The out of date links are removed and new ones
generated. Adding or removing extra pages from the document is achieved
by adding or removing their file names from the contents file.


3.3. Hierarchical Structuring
-----------------------------
The hierarchical nature of the WWW document can be utilised in the
directory organisation of its HTML files. For instance, the humour
files 'streets.html' and 'nono.html' can be placed in a 'humour'
subdirectory. curl can search over a directory tree for the HTML files
named in the contents file.


3.4. Recording of Keywords, Figures and Tables
----------------------------------------------
The files being processed can contain keyword information, and figure
and table captions. curl collects these into three special HTML files
and builds links between them and their original files.


3.5. Link Style Files
---------------------
The format of the automatically generated links is defined in style
files, allowing the style to be modified.  By default, links are added
both to the top and bottom of a file, and so there are separate style
files for these two cases.

Link styles can be applied to a subset of the pages of a document,
allowing different subsections of a document to be linked differently.

Links can be represented either by text (e.g. Next, Up) or by
user-specified icons.


3.6. A Search Engine
--------------------
Every document has its own search page. Given a set of search strings,
this page invokes a CGI script (written in C for speed) which searches
over the HTML pages of the document. It dynamically generates a page
of links to the pages containing some or all of the search strings.


3.7. A New Additions Page
-------------------------
New pages can be marked in the contents file with a '*', which causes a
link to the new page to be added to a 'New Pages' HTML file. This
permits a frequent user to discover new pages without a lengthy
search.

In addition, any contents lists which refer to a new page will be
marked with the new.gif icon or the word 'NEW!'.


3.8. Easy Removal of curl-generated Links
-----------------------------------------
The links added by curl to the HTML files can be quickly removed by
running an application called decurl. This causes the HTML files to
return to their original form.


3.9. Tools for Analysing Word Occurrences
-----------------------------------------
The curl release includes utilities for collecting information
about how frequently words occur in the pages of a
document. This is useful for deciding on a choice of keywords
utilised in pages, and also for refining the document hierarchy. For
instance, pages that make frequent references to the same words should
perhaps be grouped together.


3.10. Tools for Manipulating Links Outside the Document Hierarchy
-----------------------------------------------------------------
The document hierarchy is a useful structuring tool, but is not always
entirely sufficient.  The curl distribution includes utilities
for linking and delinking related pages without disrupting their
position in the document hierarchy.


