Integrating LaTeX3 syntax

[Rmano]

This ticket is related to #1781 about adopting a syntax highlighter specific to vimtex.

Since February 2020 (more or less) the LaTeX3 programming layer (expl3) has been incorporated in the default LaTeX format (at least for pdflatex, lualatex and xelatex). It is used more and more and, for people that are programming packages, it is quite the layer to use nowadays.

Syntax highlight of expl3 code snippets is quite awful:

There is a plugin available at https://github.com/wtsnjp/vim-expl3 that makes things quite better:

But it has the problems that:

• it has to be switched manually (set ft=expl3)
• it acts on the whole file, and the highlighting of the "normal" part of the document is better handled with the existing highlighter.

The expl3 syntax should be enabled only:

1. locally, for code enclosed in \ExplSyntaxOn and \ExplSyntaxOff;
2. for the whole buffer, if the file starts with \ProvideExplPackage, \ProvideExplClass,\ProvideExplFile.

A quick-and-dirty stopgap solution I use now is having in my .vimrc:

"
" latex3 syntax highlight fix
"
nnoremap <leader>3 :syn match texStatement "\\[a-zA-Z_:@]\+"<CR>
nnoremap <leader>2 :syn match texStatement "\\[a-zA-Z@]\+"<CR>

and manually switching when needed, but at this point doing the same with the @wtsnjp package is quite better.

"
nnoremap <leader>3 :set ft=expl3<CR>
nnoremap <leader>2 :set ft=tex<CR>

[lervag]

I'll be very happy to adapt this into #1834, but it would help to understand the main differences for the latex3 syntax. Could you provide one or two simple examples (similar to your screenshots, but preferably simpler/smaller), as well as perhaps a few lines about the main differences? Then I'll add it to the todo for the PR.

[Rmano]

The main difference is that the underscore and the colon are treated as letter, basically (and the underscore is not the subscript math command); there are other differences but they are minor. For example:

\documentclass[fleqn]{article}
\usepackage{amsmath}
\usepackage{expl3}
\usepackage{xparse}
\ExplSyntaxOn
\seq_new:N \l_test_A_seq
\seq_new:N \l_test_B_seq
\seq_new:N \l_test_C_seq

\cs_new:Npn \fillmyseq #1 #2 {
    \seq_set_split:Nnn #1 {,} {#2}
}

\fillmyseq{\l_test_A_seq}{A_1, A_2, A_3}
\fillmyseq{\l_test_B_seq}{B_1, B_2, B_3}
\fillmyseq{\l_test_C_seq}{C_1, C_2, C_3}

\cs_new:Nn \__test_do: {
    \seq_show:N  \l_test_A_seq
    \seq_show:N  \l_test_B_seq
    \seq_show:N  \l_test_C_seq
}

\NewDocumentCommand\mydo {} {\__test_do:}
\ExplSyntaxOff
\begin{document}

\mydo

\end{document}

This should look more or less like this:

A nice glitch could be to have the trailing argument specifier (the :Nn or whatever --- basicall a colon, a sequence of letters, terminated by a space) with a different highlight color.

Again, this special syntax highlight should be enabled only when expl3 is active (I commented that before...).

[lervag]

Thanks. I closed the issue, but I'm adding this as a TODO for the PR - closing only means this is now a part of #1834.

[lervag]

I've added initial support now in #1834. I've assumed that the LaTeX3 regions accept the "old" latex syntax as well, and I think the current version looks OK for the sample you provided.

However, I have not added anything for for the case when the file starts with one of \ProvideExplPackage, \ProvideExplClass, or \ProvideExplFile. If I should add support for this, please give one or more (simple) examples of such files.

Note, I think the PR #1834 is getting quite stable, so feel free to test drive it and provide feedback!

[Rmano]

HI @lervag, thanks! I am a bit in an overload in my job now --- but I'll try to test it if I can in the weekend. Sorry to be of little help lately!

[lervag]

No problem. No pressure, look at it when you have time. There's no rush :)

[Skillmon]

Files will not start with \ProvidesExpl\(Package\|Class\|File\) but with some \RequirePackage. For example, take a look at siunitx (regarded as one of the role-model expl3-using packages), fontspec (being one of the older ones) or ducksay (just to give an example of a package that I wrote :)

The problem is that expl3 isn't part of the kernel for very long yet, so all those packages support "old" LaTeX kernels by explicitly loading expl3 or xparse even though this isn't necessary nowadays.

[Rmano]

One short example could be unicode-math.sty:

%%
%% This is file `unicode-math.sty',
%% generated with the docstrip utility.
%%
%% The original source files were:
%%
%% unicode-math.dtx  (with options: `base')
%% 
%% ------------------------------------------------
%% The UNICODE-MATH package  <wspr.io/unicode-math>
%% ------------------------------------------------
%% This package is free software and may be redistributed and/or modified under
%% the conditions of the LaTeX Project Public License, version 1.3c or higher
%% (your choice): <http://www.latex-project.org/lppl/>.
%% ------------------------------------------------
%% Copyright 2006-2018  Will Robertson, LPPL "maintainer"
%% Copyright 2010-2017  Philipp Stephani
%% Copyright 2011-2017  Joseph Wright
%% Copyright 2012-2015  Khaled Hosny
%% ------------------------------------------------
%% 
%%^^A%% unicode-math.dtx -- part of UNICODE-MATH <wspr.io/unicode-math>
%%^^A%% Metadata for the package code, including files and versioning

\RequirePackage{expl3}
\ProvidesExplPackage{unicode-math}
  {2020/01/31} {0.8q} {Unicode maths in XeLaTeX and LuaLaTeX}
\sys_if_engine_luatex:T
  {
    \RequirePackageWithOptions{unicode-math-luatex}
    \endinput
  }
\sys_if_engine_xetex:T
  {
    \RequirePackageWithOptions{unicode-math-xetex}
    \endinput
  }
\msg_new:nnn {unicode-math} {unsupported-engine}
  { Cannot~ be~ run~ with~ \c_sys_engine_str!\\ Use~ XeLaTeX~ or~ LuaLaTeX~ instead. }
\msg_error:nn {unicode-math} {unsupported-engine}
\endinput

[Skillmon]

I've assumed that the LaTeX3 regions accept the "old" latex syntax as well

Good assumption. Again as an example I'd pick siunitx. Fourth line of code is

\@ifpackagelater { expl3 } { 2020/01/12 }

This directly follows \ProvidesExplPackage. The package uses other non-expl3 syntax here and there as well.

[Rmano]

@lervag could you post a couple of lines on top of PR #1834 telling how to test it? I remember there was a variable to set and a thing to do about branches in Plug...

[lervag]

@lervag could you post a couple of lines on top of PR #1834 telling how to test it? I remember there was a variable to set and a thing to do about branches in Plug...

Done. I'll respond to the new comments later today/this week.

[Rmano]

Done. I'll respond to the new comments later today/this week.

Thanks! Where? I can't find it (probably my bad). No hurry, really...