Skip to content

Page 1

ND 10898 SIFT Information Retrieval System

INTRODUCTION

SIFT is a general information retrieval system allowing searching in free text. Documents may be included with their full text or in the form of abstracts. Named fields can be defined in various ways, allowing the system also to handle forms oriented documents. The database part is designed in such a way that even very large document collections may be searched and updated rapidly. The query language is general and powerful, and the simpler functions are very easy to use.

FEATURES

  • Searching by Boolean combinations of words (and, or, not)
  • Searching by word distance (adjacent words as a special case)
  • Searching for words in the same sentence or paragraph
  • Searching for truncated terms
  • Searching on field values
  • Questions and results stored for later use
  • Search macros
  • Searching of several data bases in parallel
  • Quick browsing through retrieved documents
  • Highlighting of search terms
  • Documents may be displayed in user-defined formats
  • User-defined fields and document layout
  • Parts of documents may be defined as not searchable
  • Parts of documents may require a password to be searched or displayed
  • Access to databases and single documents may be restricted
  • Use of special commands may be restricted

APPLICATIONS

SIFT covers a very broad range of applications. It is suited for any large collection of "documents" (which may be real documents, reports, abstracts, descriptions of objects, etc.) that should be retrievable by their contents. Some examples are:

  • Library services
  • Legal documents
  • Office archives
  • Catalogues
  • Publishing
  • Newspapers

PRODUCT DESCRIPTION

COMMAND LANGUAGE, HELP FUNCTIONS

SIFT is operated through a command language with some similarities to that of the SINTRAN operating system. However, the inexperienced user does not even need to be aware of the existence of all the possible parameters, as the system assumes default values for all remaining parameters when a command is terminated.

All meaningful keys of the NOTIS keyboard are implemented, and the last command line may be edited.

Use of the HELP command (or 'HELP' key) results in a description of each command and its use. HELP can also be used to obtain explanations of each parameter. HELP is accepted as a response to any prompt from the system, and will result in a detailed explanation of the answers or parameters that may be given at that point. The word STOP (or use of the 'CANCEL' or 'HOME' keys) will also always be accepted, resulting in a return to the command prompt with no action.

DATABASE ADMINISTRATION

Databases are defined by the DEFINE DATABASE command. Among the attributes that can be defined for each database are:

  • Structure of references in the dictionary file
  • Password for this database

'Stop words' (i.e. common words which have no relevance for searching, but constitute a great part of the text) may be defined for each data base in order to reduce the size of the dictionary file.

Fields and their attributes are defined by DEFINE FIELD. Attributes such as length, type (conversion), security level, etc., may be defined.

For each database several document types may be defined by the DEFINE TYPE command. A document type specifies attributes like:

  • Which fields to include
  • How the fields are identified in the input documents
  • The character sequence(s) that separates documents
  • Paragraphs
  • Sentences (normally '. ')
  • Words (normally a space)
  • Document parts which are not to be searchable
  • Document parts with restricted access
  • Type of text storage (in internal SIFT files or as SINTRAN 'flat' files)

It should be noted that the DEFINE FIELD and DEFINE TYPE commands may be used to define a wide variety of document layouts. However, it is always assumed that the documents are prepared by some other system (like NOTIS-WP or a data entry system). SIFT does not include a data entry or editing part.

Page 2

Updating

Transactions for updating are generated by the commands ADD (to add a new document or a file to the database), EDIT (change a retrieved document) using, e.g. NOTIS-WP, DELETE (delete retrieved document(s)).

The updating process itself is activated by the command UPDATE. The present version requires that the database is closed for searching while the updating process is going on.

SIFT does not include any rollback/recovery function, so if a crash occurs during updating, the database must be copied from the backup.

Retrieval

Retrieval is performed by giving the command FIND followed by a word or several words combined using the SIFT query language.

The operators of the query language are:

  • AND, OR, ANDNOT,
  • PAR, NOTPAR,
  • SENT, NOTSENT,
  • /m,n/ (word distance),
  • . (right hand truncation),
  • IN, EQ, LT, LE, GT, GE, BETWEEN,
  • CLASS, SAME

Parentheses may be used to achieve any degree of complexity. A question may use references to earlier questions. Macros may also be defined and referred to.

Other commands used for retrieval are EXTEND, which extends the previous question, and DISPLAY MASK, which displays words starting with a specified stem. The command SELECT may be used to combine a subset of the displayed words to a query.

Up to four databases can be searched in parallel.

Display of Documents

Documents may be displayed by the SHOW command or directed to a printer by the PRINT command.

When the HIGHLIGHT mode is set, search words in the text are shown in inverse video, and when the FOCUS mode is set, only the context of the search words is shown.

Display formats consisting of selected fields may be defined by the DEFINE FORMAT command. Formats may be used in subsequent SHOW commands.

Some commands allow easy navigation forwards or backwards through result lists or databases.

Requirements

SIFT runs only on the ND-500 series of computers. SINTRAN III version J or later is required.

The disk space required to establish a text database varies from two to eight times the size of the original text, depending on the definitions and options used.

Documentation

SIFT User Guide (published by the Norwegian Directorate of Organization and Management) ND-SIFT - Appendix to "SIFT User Guide" ... ND-60.241 EN
ND-SIFT - Tillegg til "SIFT User Guide" ... ND-62.241 NO