Re: [code] [scintillua] Match patterns between embedded lexer start/end?

From: Mitchell <>
Date: Fri, 4 Oct 2013 09:51:54 -0400 (Eastern Daylight Time)

Hi Claire,

On Fri, 4 Oct 2013, Claire Lewis wrote:

> [snip]
>> The second method depends on how many possibilities there are. If
>> there
>> are only a few, say `n`, then you could duplicate your embedded lexer
>> `n`
>> times with `n` different _NAMEs and embed for each possible start and
>> end
>> combination rule. Any more than a few and you risk hitting the maximum
>> number of patterns allowed in the final grammar (~32700, imposed by
>> LPeg).
>> As you can imagine, this method is "safe" and would not rely on state.
> Hmm, this might work for this specific case, where I'm attempting to
> allow a lexer to handle variations of long-strings in Lua.
> One such application is to lex LuaJIT FFI cdef declarations:
> ffi.cdef [[
> ...
> ]]
> But I also want to allow:
> ffi.cdef [=[
> ...
> ]=]
> .. and so forth. I'd be happy enough to limit it to a few versions.
> Really those two are probably enough.
> The question is, how do I rename the embedded lexer? In this case I'm
> using the existing cpp.lua lexer - am I right in thinking I would have
> to replicate that multiple times, as opposed to some method where my
> "meta" lexer, which uses lua by default and embeds cpp could load the
> same lexer from cpp.lua but with multple names?

Try the attached patch. You should be able to do something like this from
your "meta" lexer:

    local cpp = l.load('cpp')
    local cpp2 = l.load('cpp', 'cpp2')

    l.embed_lexer(M, cpp, P('[['), P(']]'))
    l.embed_lexer(M, cpp2, P('[=[', P(']=]')

You need to wrap your start and end rules in tokens, but you get the idea.

If you manage to get that to work, let me know and I'll commit the patch
to hg so others can take advantage of this technique.


You are subscribed to
To change subscription settings, send an e-mail to
To unsubscribe, send an e-mail to
Received on Fri 04 Oct 2013 - 09:51:54 EDT

This archive was generated by hypermail 2.2.0 : Sat 05 Oct 2013 - 06:49:53 EDT