Incremental scanning and token-based editing

Abstract: A primary goal with this thesis work has been to investigate the consequences of a token-based program representation. Among the results which are presented here are an incremental scanning algorithm together with a token-based syntax sensitive editing approach for program editing.The design and implementation of an incremental scanner and a practically useful syntax-sensitive editor is described in some detail. The language independent incremental scanner converts textual edit operations to corresponding operations on the token sequence. For example, user input is converted to tokens as it is typed in. This editor design makes it possible to edit programs with almost the same flexibility as with a conventional text editor and also provides some features offered by a syntax-directed editor, such as template instantiation, automatic indentation and prettyprinting, lexical and syntactic error handling.We have found that a program represented as a token sequence can on the average be represented in less than half the storage space required for a program in text form. Also, interactive syntax checking is speeded up since rescanning is not needed.The current implementation, called TOSSED - Token-based Syntax Sensitive Editor, supports editing and development of programs written in Pascal. The user is guaranteed a lexically and syntactically correct program on exit from the editor, which avoids many unnecessary compilations. The scanner, parser, prettyprinter, and syntactic error recovery are table-driven and language independent template specification is supported. Thus, editors supporting other languages can be generated.