Re: [code] Oddity in Rexx syntax colouration

From: Mitchell <>
Date: Wed, 16 Dec 2015 10:30:47 -0500 (EST)

Hi Michael,

> I've come into a bit of a problem with the Rexx lexer. The problem is that
> it *is* a lexer. And lexing isn't enough for properly highlighting Rexx
> code.
> As an example of what I mean, *every* word in the following code is an
> identifier in some circumstances and a keyword in others:
> /**************************************** 'do' is a
> variable***************************************/do = 1say
> do/**************************************** 'do' is a keyword* 'while'
> is a variable***************************************/do while = 1 to 5
> say whileend/**************************************** 'do' is a
> keyword* 'while' is a keyword* 'forever' is a
> variable***************************************/forever = 1do while
> forever <= 5 say forever forever = forever +
> 1end/**************************************** 'do' is a keyword*
> 'forever' is a keyword* 'while' is a
> variable***************************************/while = 1do forever
> if while > 5 then do leave end say while while = while + 1end
> This means that when I use the (newly-corrected) Rexx lexer, each one of
> those words is marked in blue as a keyword, even though this is not always
> the case.
> Is it feasible to make a smarter "lexer" that can tell the difference
> between "do" as a statement with "while" as a keyword on it in one context
> while having do as a variable in another and while as a variable in yet
> another?

You can store a per-line state value to keep track of status that needs to
persist between lexings. This is the undocumented `buffer.line_state`
table, with line number keys (starting at zero) and integer values.
Despite the fact it is a buffer property, lexers have access to it
(perhaps this is not the best idea?). As the lexer encounters state
"boundaries", it should update `buffer.line_state` (via lpeg.P function
that returns its index argument to signify a match). In theory, if you can
map a word's "type" to a state, then you should be able to do what you're
trying to upon encountering those words, but I haven't done anything like
this myself.

I hope this is somewhat clear.


You are subscribed to
To change subscription settings, send an e-mail to
To unsubscribe, send an e-mail to
Received on Wed 16 Dec 2015 - 10:30:47 EST

This archive was generated by hypermail 2.2.0 : Thu 17 Dec 2015 - 06:50:03 EST