Including package luacode causes some legit brackets to mismatch

[poetaman]

This was hard to narrow down!

Including package luacode causes some legitimate brackets to show as if they are unbalanced/unmatched/mismatched.

command

vim -u minimal.vim minimal.tex

minimal.vim

set nocompatible
let &runtimepath  = '~/.vim/plugged/vimtex,' . &runtimepath
let &runtimepath .= ',~/.vim/plugged/vimtex/after'
filetype plugin indent on
syntax enable

minimal.tex
Comment the line \usepackage{luacode} and load vim a second time to see that the problem doesn't occur if luacode is not included.

\documentclass[notitlepage,12pt]{article}
\usepackage{luacode}
\newcommand{\somecommand}{
\loop
\begingroup%
    \directlua{loopcnt=loopcnt+1}%
	\ifnum\somecount=1%
		\directlua{prevcount=prevcount+\luastring{\pgfmathresult}}%
	\else%
		\directlua{prevcount=prevcount+\luastring{\someamt}}%
	\fi%
\endgroup%
\if\someothercnt0%
\repeat%
\directlua{linecount=math.floor(prevcount)+1;}%
}

screenshot

[lervag]

Can you explain this code?

\directlua{prevcount=prevcount+\luastring{\pgfmathresult}}

Is the entire string prevcount=prevcount+\luastring{\pgfmathresult} to be interpreted as luacode? That is, is {\pgfmathresult} supposed to be a lua table block?

[poetaman]

@lervag Yes the entire string prevcount=prevcount+\luastring{\pgfmathresult} is valid lua code. Macros \luaescapestring of LuaTeX and \luastring of package luacode (which just 'wraps' \luaescapestring as \luastring) convert the contents of its argument (which is supposed to be a valid TeX token sequence) to a valid Lua string. Then Lua does its magic of automatic type conversion (Arithmetic operators, Coercions & conversions). So prevcount=prevcount+\luastring{\pgfmathresult} is essentially valid Lua code that increments a Lua integer held in prevcount by integer brought in as a Lua string from TeX side of world.

AFAIK, anything inside \directlua{...} should be considered valid Lua code. As a sidenote, does VimTeX syntax highlighting intend to support LuaTeX? In that case, any code inside \directlua{...}, \latelua{...}, and potentially environments like \begin{luacode}... should be highlighted with Lua syntax highlighting.

From luacode package:

From LuaTeX manual:

From Lua manual:

and...

[lervag]

@lervag Yes the entire string prevcount=prevcount+\luastring{\pgfmathresult} is valid lua code. ...

Thanks for the explanation! As you probably understand, I am not so acquinted with lua.

So, this leaves us with a quite hard problem: {..} has a specific meaning in lua, so we need to adjust the syntax rules to somehow allow/understand the nested matching of {} pairs.

I'll look into it! As a first, I will consider the \directlua command, and then we can look at the other variants after.

[poetaman]

You are welcome! Btw, official LuaTeX documentation & implementation defines only \directlua{...} & \latelua{...} pairs to contain lua code. So you might want to enable them both together, and leave the rest for later (like luacode package's environment luacode). Most people anyway just use those two (and avoid loading yet another package like luacode), at least that's my eyes observation scrolling LuaTeX forums for couple years.

For the difficulty of {...}, am new to Vim so can't say where the difficulty lies. I suppose someone will need to detect pattern \somevalidtexmacroname{...} lua syntax and color it with TeX colors or a constant single color (so it stands out from rest of lua syntax). So in our example, \directlua{prevcount=prevcount+\luastring{\pgfmathresult}}, prevcount=prevcount+ will follow usual lua syntax highlighting, and \luastring{\pgfmathresult} would either have TeX highlighting or special single-color highlighting.

[lervag]

For the difficulty of {...}, am new to Vim so can't say where the difficulty lies. I suppose someone will need to detect pattern \somevalidtexmacroname{...} lua syntax and color it with TeX colors or a constant single color (so it stands out from rest of lua syntax). So in our example, \directlua{prevcount=prevcount+\luastring{\pgfmathresult}}, prevcount=prevcount+ will follow usual lua syntax highlighting, and \luastring{\pgfmathresult} would either have TeX highlighting or special single-color highlighting.

Ah, no, the point is that the {} inside the lua code results in the } being matched with the opening { of \directlua. I.e., the final } in \directlua{prevcount=prevcount+\luastring{\pgfmathresult} will be interpreted as the } that corresponds to the opening {. To fix this, we need to somehow allow the {} pair inside the luacode, and that's what I mean is a hard problem.

By the way: Is \luastring{\pgfmathresult} TeX code within lua code? So, this is inception? That is, we have LaTeX with Lua with LaTeX? I assume you understand that this is complex?

If I understood correctly above: what is the rule about using LaTeX within the lua code?

One option would be to disable the nested lua syntax and instead just give it a generic fixed "code" highlighting (similar to \verb+...+). Much less fancy, but it would be easier to make it robust.

[poetaman]

From my understanding, \luastring{\pgfmathresult} is just syntactic sugar that makes it look like TeX code. But the exact answer to that question is not necessary, what is essential is that it stands out from rest of lua code syntax. So for instance, in the screenshot below, text in red should be colored with Lua syntax, text in blue should be colored with TeX syntax, and text in green can either be colored with one color without any complex syntax highlighting in it ( lets say entire string \luastring{\pgfmathresult} gets colored with gray), or colored with TeX syntax colors (this being more difficult I guess).

So in the simple case where entire substring \luastring{\pgfmathresult} within Lua code gets colored with one special color, it should require adding one rule to Lua Syntax while loading VimTeX: color entire regex match below with special color when Lua syntax is ON. That will avoid latex syntax <- within lua syntax <- within latex syntax. And all we would need is turning patched Lua syntax ON in \directlua{<lua code>}, and \latelua{<lua code>} blocks (which will be a TeX filetype feature I guess).

[lervag]

I just pushed an update that I think should fix the main error here. Can you update and test? I also added the texCmd group to the lua syntax, which should allow matching tex commands; let me know what you think!

[poetaman]

@lervag The original mismatch problem is gone. Syntax highlighting is almost there, few points:

1. The special syntax highlighting in \directlua{} is enabled only if package laucode is included. Actually \directlua{} and \latelua{} are defined in LuaTeX engine, and most of the LuaTeX code won't include luacode package. This highlighting should be enabled for all *.tex files. In TeX community, it is expected that people won't redefine these command names and use them for other TeX engines, so it is safe to do this.

2. \luastring's syntax highlighting is different in different parts of \directlua. This pattern \luastring can appear anywhere within Lua code where a string can be (its essentially like a function that returns string). Check the difference between \luastring inside and outside the function:

[lervag]

@lervag The original mismatch problem is gone. Syntax highlighting is almost there, few points:

Good to hear it!

1. The special syntax highlighting in \directlua{} is enabled only if package laucode is included. Actually \directlua{} and \latelua{} are defined in LuaTeX engine, and most of the LuaTeX code won't include luacode package. This highlighting should be enabled for all *.tex files. In TeX community, it is expected that people won't redefine these command names and use them for other TeX engines, so it is safe to do this.

Well, this is not that simple. Loading the Lua syntax does come with a small cost, and if I do that for everyone regardless of whether it is relevant this cost will impact all users.

Still, I see your point, and so we need to find a middle ground that works well. I see several paths.

• Load a generic version without nested syntax as default. Load nested syntax if the luacode package is detected. Note: One may always load the extra luacode syntax by specifying the g:vimtex_syntax_packages option.

• Parse the current file for \directlua command and load the nested syntax if it is found (this might be possible if we can do it at a time where we already have the buffer content easily available).

2. \luastring's syntax highlighting is different in different parts of \directlua. This pattern \luastring can appear anywhere within Lua code where a string can be (its essentially like a function that returns string). Check the difference between \luastring inside and outside the function:

This is because the \luastring{...} is detected as a texCmd element; however, within the function block, the texCmd is no longer allowed to match. I'm not quite sure exactly how to "fix" this, because it looks like an endless rabbit hole. You will probably always be able to find an example that doesn't look perfect. This is an intrinsic issue when we work with nested/coupled syntaxes.

So, what is the best solution here? One possibility is to attempt to hack again and allow texCmd within the luaFunctionBlock element.

PS! I notice your lua code looks different than mine, so I'm curiuos if you load a different lua syntax plugin than I do. If so, which plugin do you use?

[poetaman]

@lervag Ah, if just requires adding g:vimtex_syntax_packages would do the trick then I think that's makes sense. That way people who expect to come across luatex files often can set, or don't mind the additional cost, can put it there... The cost that you mention is a one time cost (like during load time), or repeating cost?

Regarding inconsistent syntax colors for \luastring{...} in different scopes of Lua code, one other option that comes to mind is to add this pattern to String highlight group. So wherever a string is colored, this gets colored with same color. This is intuitive as the \luastring always returns a Lua string, so might as well color that. Am not a vim expert, though that way I think you won't have do define another syntax token type, and take care of all of its scopes too.

I use lua syntax from plugin: https://github.com/tbastos/vim-lua. There is another one, though the description says they have disabled support for lower versions of Lua: https://github.com/euclidianAce/BetterLua.vim. Though don't know if that affects 5.1, which not be a great idea to not support as luaJIT supports only Lua 5.1, and a lot of the Lua users like their code to be Lua 5.1 compliant code.

[lervag]

@lervag Ah, if just requires adding g:vimtex_syntax_packages would do the trick then I think that's makes sense. That way people who expect to come across luatex files often can set, or don't mind the additional cost, can put it there... The cost that you mention is a one time cost (like during load time), or repeating cost?

Yes, one time cost (during load); per session. But I do think this is a very minimal cost, so you probably won't care.

Regarding inconsistent syntax colors for \luastring{...} in different scopes of Lua code, one other option that comes to mind is to add this pattern to String highlight group. So wherever a string is colored, this gets colored with same color. This is intuitive as the \luastring always returns a Lua string, so might as well color that. Am not a vim expert, though that way I think you won't have do define another syntax token type, and take care of all of its scopes too.

I'm sorry, but I did not understand that. What do you mean "add this pattern to String highlight group"; why would that solve this issue?

Let's settle this one before we leave this issue.

I use lua syntax from plugin: https://github.com/tbastos/vim-lua. There is another one, though the description says they have disabled support for lower versions of Lua: https://github.com/euclidianAce/BetterLua.vim. Though don't know if that affects 5.1, which not be a great idea to not support as luaJIT supports only Lua 5.1, and a lot of the Lua users like their code to be Lua 5.1 compliant code.

Thanks!

[lervag]

I've implemented a hack that seems to work. Let me know if you find more edge cases.