Re: [code] partial lexing again

From: Mitchell <>
Date: Wed, 23 Oct 2013 09:15:37 -0400 (EDT)

Hi Cosmin,

On Wed, 23 Oct 2013, Cosmin Apreutesei wrote:

> Hi,
> I tried to test the idea we discussed[1] for partial lexing today.
> Basically I tried to call the hypertext lexer with different pieces of
> the same text, always starting at whitespace, to see if it returns the
> same tokens every time. It seems that at least the html parser is more
> context sensitive than that (i.e. starting at whitespace is not enough
> to guarantee that the same tokens will be returned).
> For example, the string `<script type="text/javascript">` breaks into:
> <
> script
> type
> =
> "text/javascript"
> But the string ` type="text/javascript">` breaks into:
> type="text/javascript">
> Is this expected? If yes, then I'm afraid I haven't understood the
> idea behind partial lexing very well. Care to explain this again?

You're right that for at least the HTML lexer whitespace is not enough
when you're inside tag elements. I need to fix this. For simpler lexers,
whitespace is always enough. Sorry for the confusion and trouble you've
been having :(


You are subscribed to
To change subscription settings, send an e-mail to
To unsubscribe, send an e-mail to
Received on Wed 23 Oct 2013 - 09:15:37 EDT

This archive was generated by hypermail 2.2.0 : Thu 24 Oct 2013 - 06:32:42 EDT