Re: [code] partial lexing again

From: Cosmin Apreutesei <>
Date: Thu, 24 Oct 2013 16:15:22 +0300

> You're right that for at least the HTML lexer whitespace is not enough when
> you're inside tag elements. I need to fix this. For simpler lexers,
> whitespace is always enough. Sorry for the confusion and trouble you've been
> having :(

Hi Mitchell, thanks for answering. So, for the hypertext lexer at
least, I would have to find out if the space is inside a tag and if so
go back to the "tag" styled position. This is easy, the problem is
that the whitespace tagging trick is not a reliable way to know the
language of every token. Consider the case below:

here i'm in hypertext language<script type="text/javascript"> these
whitespaces tell me I'm in javascript language
</script><this-is-a-html-tag but for all I know I'm still in
javascript lang>

I think the list of tokens returned by the lexer need to contain
markers for the beginning and end of each embedded language. This
could be useful for other purposes as well (eg. select or highlight
all javascript code etc).

You are subscribed to
To change subscription settings, send an e-mail to
To unsubscribe, send an e-mail to
Received on Thu 24 Oct 2013 - 09:15:22 EDT

This archive was generated by hypermail 2.2.0 : Fri 25 Oct 2013 - 06:42:46 EDT