RFC: Adopting full syntax support into vimtex

[lervag]

As many may know, vimtex does not provide its own syntax plugin. Instead, it builds on top of the one provided by Dr. Chip which is shipped with both Vim and neovim.

I've started to think about changing this. As a first step, I would adopt the latest version by Dr. Chip and rewrite it in terms of vimtex code style. Through some iterations of improvements, I think this could lead to a reduced workload with respect to maintenance and further development. Pros and cons of the top of my head.

Pros:

1. Having full control means we can merge and optimize the implementation.
2. Having full control means we can fix issues raised here that are not really vimtex issues.
3. Can add more tests for syntax related features to assist in controlled development.
4. I may learn more about syntax scripts.
5. May be possible to fix some sync related bugs and similar. With time.
6. It would mean that vimtex becomes a full filetype plugin in every sense, i.e. no clash with internal plugins.

Cons:

1. More work for me.
2. More area where there can be failures.
3. Would need to consider some cases like whether g:tex_flavor should be respected/overwritten and similar.
4. Unsure about the license of tex.vim and whether it is OK to "fork" it.

My thoughts for how to do it is something like this:

• Copy latest version.
• Clean up code (non breaking), adapt option names into vimtex schemes (e.g. g:tex_conceal into something like g:vimtex_syntax_config = {'conceal': ...} and so on).
• Remove some legacy stuff (e.g., I think perhaps things like g:tex_fast is unnecessary, similar with g:tex_nospell and perhaps also g:tex_conceal).

In any case, I would be happy to hear other peoples thoughts and opinions on this.

[clason]

I would welcome this (not that Dr Chip isn't doing a great job!) and couldn't think of a person I'd trust more. Con 1 is of course a big con and would understandably be sufficient in itself.

The possibility of faster updates would indeed be appealing -- there was a recent nasty bug in the syntax file that took quite a long time to percolate down to neovim. (Of course, it's trivial to put a fixed file into .config/nvim/syntax, but still.)

Another advantage is that you can be more opinionated about defaults than a default vim syntax -- for example, finally making latex the default TeX flavor. Now that you target vim8, you'd also be free to rely on more recent vim features should those improve either performance or maintainability (although I'm not sure how much room for either there is in a syntax file).

[adigitoleo]

Hi,

Firstly, let me just say that I appreciate this plugin very much. I was able to write my honours thesis in neovim and it was awesome.

Now, back on topic, I have noticed one issue in the syntax highlighting for a while, but it was not a big deal so I didn't worry. I came here to post an issue, but maybe I will piggyback in this thread first, since I am not sure now if it is a problem in vimtex or in the syntax file.

When I place the cursor on a begin{<env>} (or end{<env>}), the the current line as well as the matching \end (or \begin) are highlighted (I think using hi Search?). Now, when I switch to the next/previous buffer the highlights remain visible on those lines. In other words, these environment highlights are not local to the tex buffer or are not cleared when leaving. It seems to be the same regardless of set hidden.

Does vimtex draw these highlights? Or should I go to the syntax maintainer (Dr Chip?). It should be trivial to reproduce, so I could post a proper issue if you prefer.

Since I am here, I will give a quick vote on full syntax support: Yes. I agree with @clason that such a framework (even as an opt-in) could be useful. I am very green with LaTeX, but it seems there are many versions/variations and people could hopefully help to add support for their favourite setup. For example, I would welcome support for TikZ and friends.

[Rmano]

+1; notice that just looking around I found this: https://github.com/gi1242/vim-tex-syntax (not tested).

Another really nice thing would be to add LaTeX3 syntax highlighting; it would be activated between \ExplSyntaxOn and \ExplSyntaxOff --- right now it's a bit a pain in the back:

(I'll copy here the file if somebody is interested in giving it a try; notice that the syntax highlight on GitHub is also off, this is quite new stuff)

There is a "dirty hack" here, that I use manually: https://tex.stackexchange.com/questions/44624/vim-configuration-for-expl3

% see https://tex.stackexchange.com/questions/557511/how-to-query-iffilehaschanged
\documentclass{article}
\usepackage{expl3}
\ExplSyntaxOn
    \str_new:N \g_myfile_name_str
    \str_gset:Nn \g_myfile_name_str {myfile.txt}
    \str_new:N \g_myfile_old_mdfive_str
    \str_new:N \l_myfile_mdfive_str
    \cs_new_protected:Npn \getmdfive % get hash is not expandable, so protect
    {
        \file_get_mdfive_hash:nN {\str_use:N \g_myfile_name_str} \l_myfile_mdfive_str
        \str_use:N \l_myfile_mdfive_str
    }
    \cs_new:Npn \getoldmdfive
    {
        \str_use:N \g_myfile_old_mdfive_str
    }
    \cs_new:Npn \IfMyfileChanged #1 #2
    {
        \str_if_eq:NNTF \g_myfile_old_mdfive_str \l_myfile_mdfive_str {#2} {#1}
    }
    \AtEndDocument
    {
        \iow_now:cx { @auxout }
        {
            \token_to_str:N \ExplSyntaxOn
            \iow_newline:
            \str_gset:Nn \token_to_str:N \g_myfile_old_mdfive_str {\str_use:N \l_myfile_mdfive_str}
            \iow_newline:
            \token_to_str:N \ExplSyntaxOff
        }
    }
\ExplSyntaxOff
\begin{document}

MD5 was \getoldmdfive

MD5 is now \getmdfive

Changed? \IfMyfileChanged{Yes, it has changed}{No, it's the same as before}

\end{document}

[clason]

Another thing that this would allow is cleaning up and consolidating (or extending, if necessary) the highlight groups. I've been thinking of finally biting the bullet and writing my own tex-specific color scheme and ideally would like to color math commands differently from text commands (and differently from environments and labels etc.), and this would be a good opportunity. (Let me know if discussion of tex color schemes would make for a good separate issue.)

If you do go for this, I'd second @Rmano's suggestion of looking at https://github.com/gi1242/vim-tex-syntax as a basis, which looks quite clean.

Also, @Rmano: did you see https://github.com/wtsnjp/vim-expl3?

[Rmano]

Also, @Rmano: did you see https://github.com/wtsnjp/vim-expl3?

No, I was not aware... is it compatible with vimtex? I really do not know how to use it alognside the normal latex. It should be a sublanguage, triggered by \ExplSyntaxOn and ..Off or the expl3 sprecific include file (I do not remember how to check them, but for sure not by the extension).

[clason]

I would assume it won't conflict so you can just load it, but I don't know the best way to inject it into a tex syntax; I'd look into how vimtex extends the builtin syntax for the different packages in https://github.com/lervag/vimtex/tree/master/autoload/vimtex/syntax.

In any case, Expl3 support should probably be a separate issue, independent of whether the base tex syntax file is replaced.

[lervag]

@clason @Rmano @adigitoleo Thanks for the support/votes! I think I want to try this, but as said and as I think you understand, it will not be something I'll be doing first thing. Perhaps I'll start working on it and finishing a version before Christmas, but I won't make promises.

@clason

... not that Dr Chip isn't doing a great job!

Hear hear!

Another advantage is that you can be more opinionated about defaults than a default vim syntax -- for example, finally making latex the default TeX flavor.

Yes, this is one of the reasons I want to do this. I think and hope that it is possible to both simplify and "standardise" more on what I think are more sensible and modern defaults.

(Let me know if discussion of tex color schemes would make for a good separate issue.)

It might be. I'm not 100% sure what you mean, and I don't mind discussing things. I would prefer to keep the topics separated, so let's do it in a separate issue.

@adigitoleo

Firstly, let me just say that I appreciate this plugin very much. I was able to write my honours thesis in neovim and it was awesome.

Thanks! :D

I came here to post an issue, but maybe I will piggyback in this thread first, since I am not sure now if it is a problem in vimtex or in the syntax file.
...

The feature you describe sounds like the matchparen feature, see :help g:vimtex_matchparen_enabledand alsovimtex-nf-enhanced-matchparen`. If you want to further discuss this, please open a new issue for it.

@Rmano

Another really nice thing would be to add LaTeX3 syntax highlighting; it would be activated between \ExplSyntaxOn and \ExplSyntaxOff --- right now it's a bit a pain in the back:

Interesting! But for my own sanity and for keeping topics separated, I would prefer if we discuss this in a new/other thread. It seems @clason found a relevant plugin. If it does not immediately work with vimtex, feel free to open an issue and we can discuss it further there.

@Rmano and @clason

... I found this: https://github.com/gi1242/vim-tex-syntax (not tested).

Thanks for the heads up!

If you do go for this, I'd second @Rmano's suggestion of looking at https://github.com/gi1242/vim-tex-syntax as a basis, which looks quite clean.

I agree that it at looks clean. I'm worried, though, that it would require a more full cleanup of all the syntax stuff. Has anyone tried to use vim-tex-syntax with vimtex?

---

When I start work on this I will definitely work in a separate branch that will be merged as a PR. The change will be significant, and so I think it might require a major version bump when the PR is merged. Thus I also don't want to merge until I'm confident that users will

1. not really notice any major changes, or
2. recognize the change as an improvement.

I will be very happy if I get help with testing when things are progressing. I would also like feedback on which of the old/legacy features are important to keep and which can be "safely" removed.

One relatively important aspect is efficiency. Thus, it may be very useful to be able to test the speed of the syntax before and after. I want to simplify things and remove e.g. g:tex_fast, but this means I must be sure that the script is actually fast enough! If anyone has a good idea for how to test this that would be much appreciated.

[clason]

It might be. I'm not 100% sure what you mean, and I don't mind discussing things. I would prefer to keep the topics separated, so let's do it in a separate issue.

👍 (it does have some bearing on this, since it involves the highlight groups that are defined in the syntax file and how they are linked, but it makes sense to first discuss what can be done with the current file).

I agree that it at looks clean. I'm worried, though, that it would require a more full cleanup of all the syntax stuff. Has anyone tried to use vim-tex-syntax with vimtex?

I've briefly put it in my config/nvim/syntax/ directory, and nothing broke -- some things looked better (with my color scheme), some worse, and I noticed about a 60% reduction in startuptime (for the syntax file only!)

When I start work on this I will definitely work in a separate branch that will be merged as a PR. The change will be significant, and so I think it might require a major version bump when the PR is merged. Thus I also don't want to merge until I'm confident that users will
...
I will be very happy if I get help with testing when things are progressing. I would also like feedback on which of the old/legacy features are important to keep and which can be "safely" removed.

Might I suggest then instead to develop on the main branch but put the new syntax behind a (by default off) option g:vimtex_new_syntax? Then if you're happy, you can just flip this to default on (and bump the version). This way you get the development version under more eyes and make it easier for us to do A/B testing (or turning it off temporarily if it breaks for a while).

(Assuming this is even feasible.)

One relatively important aspect is efficiency. Thus, it may be very useful to be able to test the speed of the syntax before and after. I want to simplify things and remove e.g. g:tex_fast, but this means I must be sure that the script is actually fast enough! If anyone has a good idea for how to test this that would be much appreciated.

Definitely (you know me ;))! Another aspect would be to allow using "high-level" syntax groups to be used in ex commands -- for example, replace only in math zones, or outside math zones (but not commands) etc.

[Rmano]

Interesting! But for my own sanity and for keeping topics separated, I would prefer if we discuss this in a new/other thread. It seems @clason found a relevant plugin. If it does not immediately work with vimtex, feel free to open an issue and we can discuss it further there.

Done!

[Rmano]

Might I suggest then instead to develop on the main branch but put the new syntax behind a (by default off) option g:vimtex_new_syntax? Then if you're happy, you can just flip this to default on (and bump the version). This way you get the development version under more eyes and make it easier for us to do A/B testing (or turning it off temporarily if it breaks for a while).

(Assuming this is even feasible.)

If it's feasible, I agree. Another possibility is to define a syntax like tex+ and then switch manually during the development, with set ft, and rename when ready...

[adigitoleo]

@lervag Thanks for the pointers, you are correct my issue is related to matchparen and it's not specific to vimtex.

I will keep an eye on the syntax side of things and help if/when I can, although I can't say I'm very experienced in vim-fu (or development in general 😄 )