Integration with proofreading tool

[matze-dd]

Hi,

direct usage of a proofreading tool (for spell, grammar, and some style checking) in a LaTeX editor can be comfortable. In order to avoid too many false positives from the proofreader, it should be fed only with plain text extracted from the current LaTeX source. On the other hand, locations of problems reported by the tool should be mapped back to the corresponding positions in the LaTeX source.

May I draw your attention to the project YaLafi? This is a LaTeX filter suitable for integration with an editor and a proofreading software like LanguageTool. Initially, filtering has been based on recursive regular expressions (project Tex2txt), but now it uses a more powerful scanner / parser for LaTeX analysis.

As other tools like TeXtidote and vscode-ltex, YaLafi provides mapping of character positions between the LaTeX text and the plain text seen by the proofreading tool. Therefore, messages from the proofreader can be properly related to the corresponding parts of the LaTeX text. Currently, it can be used with Vim plug-in vim-grammarous (see here) and Emacs plug-in Emacs-langtool (see here).

I do my own LaTeX writing with "plain Vim", in a rather simple, Bash-script based environment. The above filter project grew out of the desire to use LanguageTool for the check of already existent LaTeX document trees. Therefore, I wanted to make it flexible and not to produce too many false positives from the proofreading tool. Now it is in a status that one might use it in a larger framework.

Would it be worth to put some effort into integrating vimtex with a LaTeX filter, an interface like vim-grammarous, and LanguageTool? In order to keep things simple for the user, application via Docker seems sensible to me (combined usage of Java, LanguageTool, LaTeX, Python, Vim, ... in alphabetic order). Before starting work, it would be nice to know some opinions on such an undertaking.

Matthias

PS. A similar post has already been placed at the closed Issue #1311. Sorry for any inconvenience.

[matze-dd]

Sorry, one more thought.
The [ALE] documentation suggests that this might be an alternative to interfacing like in the plug-ins [vim-grammarous] or [vim-LanguageTool]. At least a simple status line as in this [Emacs example] should be possible, and highlighting of problematic text parts is apparently supported.

The main question remains: Does it make sense to integrate all these components that are based on different tools / programming languages? Or should one wait until all is available in one large integrated application (cf. TeXStudio)?

[lervag]

I'll give a more detailed comment later. But first: did you read the discussion in #1311? The relevant part of the current docs can be found in :help vimtex-nf-linting, where I propose a sort of equivalence between linting of the code and spell/grammar checking. My current stance on the topic is that these features should be handled by a dedicated plugin such as Ale. Still, I'll read your thoughts and also the links on YaLafi and think this through, and we'll have a discussion before we decide anything.

[matze-dd]

Thank you for the fast first reply. The fascinating world of Vim plugins is rather new to me.

The discussion in #1311 and the documentation in vimtex-nf-linting seem to show that there are roughly two groups of linters. One is concerned with the LaTeX synatx istself. These linters do not or only little consider the "real" text of the author. Tools from the other group try to analyse the latter text, but do not remove LaTeX's markup. They therefore tend to produce false positives.

Browsing (too lately) the vimtex documentation, I found that an interface to TeXtidote is already provided. This is a linter that analyses the "real" text and takes care of the markup. As can be seen on the [GitHub page], many people are happily using this tool. With my own LaTeX documents, it does produce too many false positives, however (LaTeX filtering is based on rather simple regular expressions).

Some (not many) people already have downloaded YaLafi or its predecessor Tex2txt. This indicates that more complex LaTeX parsing is useful at least for certain texts. As one should expect more open-source proofreading software in the future, an independent filtering tool might be practical.

The view that an own ALE-based plugin would be best becomes more and more reasonable to me. I hope that the Python code of the LaTeX filter could be a direct part of a new ALE linter component. This would avoid the need to manually install a Python package. Additionally, one could "automatically" download the LanguageTool software, as is done by [vim-grammarous]. This way, the plugin had full control over paths to its components. A user only had to install the new ALE component, and to make sure that Java and Python are present. Hopefully, this would be compatible with a plugin like vimtex.

[lervag]

Thank you for the fast first reply. The fascinating world of Vim plugins is rather new to me.

No problem, and welcome to this part of the galaxy :)

Some (not many) people already have downloaded YaLafi or its predecessor Tex2txt. This indicates that more complex LaTeX parsing is useful at least for certain texts. As one should expect more open-source proofreading software in the future, an independent filtering tool might be practical.

Definately, I agree!

The view that an own ALE-based plugin would be best becomes more and more reasonable to me. I hope that the Python code of the LaTeX filter could be a direct part of a new ALE linter component. This would avoid the need to manually install a Python package. Additionally, one could "automatically" download the LanguageTool software, as is done by vim-grammarous. This way, the plugin had full control over paths to its components. A user only had to install the new ALE component, and to make sure that Java and Python are present. Hopefully, this would be compatible with a plugin like vimtex.

One goal of vimtex, since the beginning, is to avoid implementing features that are best left to more dedicated plugins. I believe this is one such example. As far as I understand, YaLafi can be considered a dedicated LaTeX linter, or at least a filter to be used with a linter. There are several Vim plugins that are dedicated servers for linting. Ale is probably the most well-known and mature of these, but others include the older Syntastic and also coc.nvim. Vimtex works well with all of these plugins.

---

As you've noticed: vimtex does have direct support for Textidote. This is implemented as a compiler (see :help compiler for more info on this feature). The idea is to set the makeprg setting and the errorformat setting so that :make runs the linter. An alternative would thus be to add a compiler that uses YaLafi to vimtex. I won't have time to do this myself in the near future, but if you want to test it, I think you should be able to figure things out by looking at the code in VIMTEX/compiler/textidote.vim. Much of it is boiler plate. You would need to define the makeprg setting and the errorformat. The latter has a nasty syntax, but it is quite well documented (see :h errorformat). If you propose a PR for this, I will gladly assist, review, and merge when it looks fine!

[lervag]

By the way: Congrats on making this tool, it looks very good and useful! I'm impressed by how this can understand e.g. the footnote text and how languagetool works on sentences broken by footnotes! Cool stuff!

[lervag]

By the way:

Would it be worth to put some effort into integrating vimtex with a LaTeX filter, an interface like vim-grammarous, and LanguageTool?

Regardless of whether you/we add a compiler directly to vimtex, or if YaLafi is only used through vim-grammarous or ALE; it would be very nice to have a short paragraph in the vimtex docs on how to add YaLafi to ones toolbox. I propose that, when the Vim interfaces have become more "stable", we add a description somewhere in the docs on how to use this.

[matze-dd]

@lervag Thank you for the time and the kind words.

... There are several Vim plugins that are dedicated servers for linting. Ale is probably the most well-known and mature of these, but others include the older Syntastic and also coc.nvim. Vimtex works well with all of these plugins.

... You would need to define the makeprg setting and the errorformat. The latter has a nasty syntax, but it is quite well documented (see :h errorformat). If you propose a PR for this, I will gladly assist, review, and merge when it looks fine!

Reading the errorformat help, it seems to me that error highlighting directly in the LaTeX text is not possible via this mechanism. At the beginning, I would therefore opt for a combination with the other plugins.

Regardless of whether you/we add a compiler directly to vimtex, or if YaLafi is only used through vim-grammarous or ALE; it would be very nice to have a short paragraph in the vimtex docs on how to add YaLafi to ones toolbox. I propose that, when the Vim interfaces have become more "stable", we add a description somewhere in the docs on how to use this.

D'accord. I will come back when the plugin interfaces have matured.

[lervag]

Reading the errorformat help, it seems to me that error highlighting directly in the LaTeX text is not possible via this mechanism. At the beginning, I would therefore opt for a combination with the other plugins.

Do I understand correctly that YaLafi is meant to be used together with other tools? It seems YaLafi depends on custom made "shell" scripts for each spelling/grammar checker (as in the example for LanguageTool); thus you get a cli tool to run on the LaTeX document that outputs stuff like this:

1.) Line 2, column 17, Rule ID: MORFOLOGIK_RULE_EN_GB
Message: Possible spelling mistake found
Suggestion: red; Rex; reds; redo; Red; Rede; redox; red x
Only few people is lazy.    We use redx colour.

This output can be parsed by the Vim errorformat, but as I said, it is not straightforward. However, it is much easier to parse this than to parse the LaTeX log files, which I do in vimtex. And so I am quite sure that it is possible to make such a compiler for YaLafi combined with a tool like LanguageTool.

However, a more general interface to YaLafi seems more difficult. It seems one would need explicit compiler scripts for each combination of YaLafi + "spell linter".

D'accord. I will come back when the plugin interfaces have matured.

Great! In the meantime, I hope it's OK that I close this issue for now. Feel free to reopen it, or preferably open a new issue when you propose updates to vimtex.

[matze-dd]

Just for clarification of a misunderstanding:

Do I understand correctly that YaLafi is meant to be used together with other tools? ...

This output can be parsed by the Vim errorformat, but as I said, it is not straightforward. However, it is much easier to parse this than to parse the LaTeX log files, which I do in vimtex. And so I am quite sure that it is possible to make such a compiler for YaLafi combined with a tool like LanguageTool.

The application script yalafi.shell of course parses the output from LanguageTool (JSON format, as this at the end the simplest variant). And using errorformat, one can present the necessary information to Vim such that it displays the error message(s) and sets the cursor to the error location in the text. But as I understand, it isn't possible to highlight the erroneous text part. For that, one would need additional information in errorformat (length or end position of problematic text part). Indeed, highlighting seems to be an important additional feature of plugins like vim-grammarous and ALE.

[lervag]

The application script yalafi.shell of course parses the output from LanguageTool (JSON format, as this at the end the simplest variant). And using errorformat, one can present the necessary information to Vim such that it displays the error message(s) and sets the cursor to the error location in the text.

For me, this is the important part!

But as I understand, it isn't possible to highlight the erroneous text part.

This is convenient, but not important.

For that, one would need additional information in errorformat (length or end position of problematic text part). Indeed, highlighting seems to be an important additional feature of plugins like vim-grammarous and ALE.

Yes, I agree it is an additional and convenient feature, but IMHO, it is not important. Of course, there are probably a lot of people who disagree with me on this. :)