[code] partial lexing again

From: Cosmin Apreutesei <cosmin.apreutesei.att.gmail.com>
Date: Wed, 23 Oct 2013 01:17:39 +0300


I tried to test the idea we discussed[1] for partial lexing today.
Basically I tried to call the hypertext lexer with different pieces of
the same text, always starting at whitespace, to see if it returns the
same tokens every time. It seems that at least the html parser is more
context sensitive than that (i.e. starting at whitespace is not enough
to guarantee that the same tokens will be returned).

For example, the string `<script type="text/javascript">` breaks into:



But the string ` type="text/javascript">` breaks into:


Is this expected? If yes, then I'm afraid I haven't understood the
idea behind partial lexing very well. Care to explain this again?

[1] http://foicica.com/lists/code/201309/1172.html

You are subscribed to code.att.foicica.com.
To change subscription settings, send an e-mail to code+help.att.foicica.com.
To unsubscribe, send an e-mail to code+unsubscribe.att.foicica.com.
Received on Tue 22 Oct 2013 - 18:17:39 EDT

This archive was generated by hypermail 2.2.0 : Wed 23 Oct 2013 - 06:45:24 EDT