Re: Next generation of lexers

From: Robert <>
Date: Tue, 27 Apr 2010 23:00:38 +0200

On Tue, Apr 27, 2010 at 4:38 PM, mitchell <> wrote:
> Rob,
> In each variable token, add a functional pattern that always matches,
> but grab the variable name and add it to a list. In a pattern before
> identifier, check if the identifier exists in your variable list, and
> if the match succeeds, match the variable token. Otherwise, don't
> match and an ident will later match. Pseudocode:
> local variable_list = {}
> local variable = token('variable', P('Var') * P(function(patt, i)
>    variable_list[patt:match('[%w_]+', i)] = true
>    return i
>  end) * l.word)
> local potential_variable = P(function(patt, i)
>    return variable_list[patt:match('[%w_]+', i)] and i or nil
>  end * token('variable', l.word)
> local identifier = token('identifier', l.word)
> _rules = {
>  ...
>  { 'variable', variable },
>  { 'potential_variable', potential_variable },
>  { 'identifier', identifier },
>  ...
> }

Thanks a lot again, I'll try this out.
>> I put lexers for Markdown, NSIS and Lilypond up here:
>> Feel free to include the lexers, if you like, they will hopefully
>> improve as I use them more. I wasn't sure if a Markdown lexer is
>> against the point of Markdown, but then I liked it.
> Do you think they're polished enough to be included in the upcoming
> release?

NSIS and Lilypond, yes.

% Lilypond
ly lilypond

nsh nsis
nsi nsis

As for Markdown, I'm not sure. If you have a look there are some not
styled tokens to get emphasis correctly styled. There might be cases
I'm not aware of because I didn't use Markdown a lot. I mostly used
the Markdown definition sources and the doc sources.
If you would prefer to include a safer version with maybe only headers
and code let me know and I'll post a changed version.
How did you write all these lexers, by the way, did you mostly just
"translate" the Scite lexers? Is there a project that collects example
source files and keyword lists for all programming languages, a

>> Some thoughts on the new docs:
>> I had bad experiences with other editors when a lexing style changed
>> theme colors so lines became impossible to read. Maybe a note could be
>> added to add back- and foreground colors together? The diff lexer is
>> an example for this, maybe the foreground color could be changed?
>> Another issue with the diff lexer: Lua comments are treated as
>> deletions. I tried some but didn't find a good solution.
>> Another idea for the Latex lexer, maybe it would be useful to turn
>> this into a Tex lexer that covers plain Tex, Latex and Context.
>> Currently there are quite a lot of keywords in the Latex lexer but it
>> is impossible to cover all packages or new ones defined with
>> renewcommand.
>> I would do a
>> local command = token('keyword', '\\' * l.word)
>> This would cover Context's keywords as well. Having Latex environments
>> in the lexer shouldn't be a problem.
>> Then I have to look some more into LuaTeX, and have the Lua lexer embedded :-)
>> I like their explanation for using Lua over x:
> Thanks a lot for your suggestions, but if you could have code,
> patches, and/or concrete text to contribute, that would be even
> better. I'm pretty strapped for time with final exams coming up and
> rather than spending time trying to understand exactly what you'd
> like, I'd prefer to see what exactly needs to be done. Normally I have
> no problems with what you're doing :) It's just this time of year
> that's difficult for me.
> Best,
> Mitchell

Sorry, if I have been vague and/or thinking aloud.
I always accept "I'll accept a patch" as an answer. :-) It's just that
sometimes there is a compatibility reason or something... I'll try to
come up with more concrete solutions.
As for the Tex lexer I was maybe hoping for feedback from Context
users if that would be a good idea.


You received this message because you are subscribed to the Google Groups "textadept" group.
To post to this group, send email to
To unsubscribe from this group, send email to
For more options, visit this group at
Received on Tue 27 Apr 2010 - 17:00:38 EDT

This archive was generated by hypermail 2.2.0 : Thu 08 Mar 2012 - 11:43:40 EST