Re: [code] [scintillua] More lexer improvements from the vis editor community

From: Mitchell <>
Date: Wed, 22 Feb 2017 10:18:46 -0500 (EST)

Hi Marc,

On Wed, 22 Feb 2017, Marc André Tanner wrote:

> On Fri, Feb 03, 2017 at 10:00:15AM -0500, Mitchell wrote:
>> Hi Marc,
>> On Thu, 2 Feb 2017, Marc André Tanner wrote:
>>> Hi,
>>> Below you find references to lexer improvements contributed to my vis
>>> editor:
>>> * spin.lua a new lexer for *.spin files by David B. Lamkins.
>>> * rc.lua a new lexer for shell scripts for the Plan 9 shell rc(1)
>>> by Michael Forney.
>>> * sml.lua a new lexer for Standard ML (*.sml, *.fun, *.sig files)
>>> by Murray Calavera.
>>> * scheme.lua was updated to reference the defined `func` tokens in
>>> the lexer's `_rules` table.
>>> * ansi_c.lua has been updated to C11 by S. Gilles, Christian Hesse
>>> and myself. Furthermore some numerical types and errno(3) constants
>>> as specified by POSIX were added.
>>> These can all be found in the ./lua/lexers directory of the vis repository:
>>> If you decide to merge these modifications, it would be nice to avoid
>>> unnecessary white space changes to make downstream merging easier.
>> Thank you. I will look into this when I have the chance.
> I noticed you started the lexer refactorings without first integrating
> the C and Scheme lexer changes, do you still plan to look at them or
> have you concluded that they are unsuitable for inclusion?

Thanks for the follow up, and I'm sorry for the lack of communication. It
certainly may appear that I have silently rebuffed your e-mail. That is
not true and I am sorry. Your lexers are still on my to-do list, but after
I finish my current set of work. I simply have not looked at them yet.

> Regarding the C lexer it would be nice if it could treat:
> - #if 0 ... #endif as a comment, this might become a bit ugly due
> to the nested nature
> - filenames following #include directives as strings

I'll make a note and follow up with you later. The first one may be a bit
tricky, but we shall see.

> I'm busy with other stuff for now, so haven't yet looked into these.
> In the meantime a bugfix for nested variables in the bash lexer was
> contributed:

Thanks for this. Again, I'll make a note.

> I noticed that you removed some lexers. Any particular reason for that?
> Did they have specific problems I should be aware of? I know that at
> least one of my users cares about APL, so I will most likely add that
> back in my repo.

No reason other than I figured it's not worth the effort in refactoring
them if they're not likely to be used. I'm surprised to hear of a user of
APL. Regardless, you can re-add the legacy lexer to your repo. I've
designed the refactoring to handle legacy lexers without issue.

> Also is there a place where I can read upon the motivation / goals of
> the refactoring? I'm not sure I agree with some of the changes (e.g.
> word_match taking a string rather than a table?). But then you have
> much more experience in Lua, I'm sure there is a good reason for it.

No, it's all in my head :) Feel free to ask questions. I'm not even 100%
sure this was/is a good idea. It's an experiment right now that I might
end up throwing away.

First I'd like to point out that one of my goals is to keep compatibility
with legacy lexers. Right now my repository is a mix of legacy lexers and
refactored lexers and my unit tests still pass, so that's a good sign.

Since compatibility is important, you can keep the table form of
`word_match()` if you want. I personally don't like the idea of creating
giant tables of keywords and having them stick around in memory. That's
why I've moved to a single string.

Some other motivations are I don't like the idea of "magic fields" (e.g.
`_rules`, `_tokenstyles`). I think the object-style of `:add_rule()` and
`:add_style()` is better practice.

> Thanks for all your work!

Thanks for your contributions, and sorry again for my lack of


You are subscribed to
To change subscription settings, send an e-mail to
To unsubscribe, send an e-mail to
Received on Wed 22 Feb 2017 - 10:18:46 EST

This archive was generated by hypermail 2.2.0 : Thu 23 Feb 2017 - 06:51:31 EST