Additional interface for a proofreading tool

[matze-dd]

Following the discussion in Issue #1864, I'd like to propose a further vimtex compiler interface to LanguageTool. Similarly to the interface via TeXtidote, see vimtex-grammar, it extracts plain text from the LaTeX source before invoking the proofreader. A prototype of the new component is described here. Depending on the nature of the LaTeX document, it may provide better flexibility and less false positives.

Before progressing, some things would have to be clarified.

• Apparently, message display via Vim's errorformat mechanism is restricted to a one-liner. What should be included? In the version above, it only contains LanguageTool's "Message" part, but rule ID and replacement suggestions are useful and can be easily added. This will often produce long one-line messages, and might be configured by Vim variables.
• Is there any need and possibility for automated tests of such an integrated tool? (Perhaps, one could use LanguageTool's online service, such that only the Python module YaLafi had to be installed.)
• I could propose a first documentation version. For check of language mistakes in the "real text", error highlighting may be considered as useful, depending on taste. Should we add some words for an interface based on ALE?
• EDIT after reading some discussions in the Issues: Since vimtex does support numerous LaTeX macros and environments, at least the most important ones should be properly processed by the LaTeX filter. How can one organise that in a structured way? As I understand, vimtex loads declarations when it detects the inclusion of a package.
  (2nd edit:) Perhaps, one could somehow use the idea in Issue #28@YaLafi?
  (last edit for now:) The vimtex code includes the two objects b:vimtex.documentclass and b:vimtex.packages. If they are accessible when starting the "compiler", we can possibly achieve a nice integration.

[matze-dd]

The proposal needs further re-thinking. For the time being, I close the issue, hoping that this is OK.

[lervag]

First, sorry for the delay in responding!

Following the discussion in Issue #1864, I'd like to propose a further vimtex compiler interface to LanguageTool. Similarly to the interface via TeXtidote, see vimtex-grammar, it extracts plain text from the LaTeX source before invoking the proofreader. A prototype of the new component is described here. Depending on the nature of the LaTeX document, it may provide better flexibility and less false positives.

I'm generally very positive to this!

• Apparently, message display via Vim's errorformat mechanism is restricted to a one-liner. What should be included? In the version above, it only contains LanguageTool's "Message" part, but rule ID and replacement suggestions are useful and can be easily added. This will often produce long one-line messages, and might be configured by Vim variables.

I am not quite sure what you mean with "restricted to a one-liner". Do you mean the entries in the quickfix window, or the error messages from the "linter tool" that are parsed through the errorformat setting?

In general, I think you should parse so that each entry in the quickfix list has i) a line number and possibly column number to specify location of error, ii) an error message that explains the error. errorformat does allow to parse multiline error messages, but the format is slightly complicated. I can help here. Note, the text in :help errorformat should help you to design a suitable output format that is easy to parse, if that is relevant.

• Is there any need and possibility for automated tests of such an integrated tool? (Perhaps, one could use LanguageTool's online service, such that only the Python module YaLafi had to be installed.)

Both yes and no. I try to add tests for the various features of vimtex, but not all features are properly tested. I'll accept and merge a PR for this feature even if it does not have automated tests, of course, but adding automated tests would of course be even better. Note, though, that we then need to handle the install of both LanguageTool and your YaLafi.

As a first step, I would say it is sufficient to provide a good example of the feature with a good LaTeX sample and a minimal vimrc file so that we can manually test the feature.

• I could propose a first documentation version. For check of language mistakes in the "real text", error highlighting may be considered as useful, depending on taste. Should we add some words for an interface based on ALE?

Yes, the docs should be updated accordingly with an explanation of the feature and how to use it. Here one could also suggest how to set up Ale for an "improved" version with more highlighting and automatic linting.

• EDIT after reading some discussions in the Issues: Since vimtex does support numerous LaTeX macros and environments, at least the most important ones should be properly processed by the LaTeX filter. How can one organise that in a structured way? As I understand, vimtex loads declarations when it detects the inclusion of a package.
  • (2nd edit:) Perhaps, one could somehow use the idea in Issue #28@YaLafi?
  • (last edit for now:) The vimtex code includes the two objects b:vimtex.documentclass and b:vimtex.packages. If they are accessible when starting the "compiler", we can possibly achieve a nice integration.

Yes, these should be available when you start the compiler. I think it is safe to use them in a compiler plugin. Both objects are defined on initialization. Note, though, that the packages list is only fully "correct" when the document is compiled and an .fls file is available. If this is not the case, then packages only contains the detected \usepackage macros, which means that nested packages are not recognized.

The proposal needs further re-thinking. For the time being, I close the issue, hoping that this is OK.

No problem. I'll be happy to consider this feature when you are ready!

[matze-dd]

First, sorry for the delay in responding!

No problem. I really appreciate that you invest the time and reply that carefully.

I am not quite sure what you mean with "restricted to a one-liner". Do you mean the entries in the quickfix window, or the error messages from the "linter tool" that are parsed through the errorformat setting?

I mean the following problem: For each detected mistake, we can provide a certain amount of information. This is not only the explanatory error message itself, but also a list of replacement suggestions from the proofreader (often useful) and the error context as seen in plain text by the proofreader. The latter may be helpful if the LaTeX filter performed some more aggressive modifications, e.g., when handling equations.

All this together may form a quite "long line" for each reported problem, and this may become confusing, even if one uses the error list in the separate quickfix window (:cw). The idea was to make that a bit configurable by auxiliary Vim variables (e.g., g:vimtex_ltyc_show_replacements, g:vimtex_ltyc_show_context), so that the user can extend / restrict the amount of information.

The errorformat syntax is not the problem. I manage to parse oneline and multiline messages from the "linter tool" and can extract the required parts to construct the quickfix messages. But i'm still unsure, whether I completely understand the full potential of the mechanism. It seems to me, that a reconfiguration, depending on the detected error and desired information detail, is only possible by changing the above auxiliary variables (and thus "automatically" the errorformat under the hood) and re-running the compiler. If it is that way, then all is OK. Just do not want to miss a possibility.

(My concerns probably have the following root. In all other plugins like ALE, one can display a subwindow that shows all information in a structured manner. Here, we only have a one-liner in the status line, or a wrapped line in the quickfix window.)

No problem. I'll be happy to consider this feature when you are ready!

Thanks again. This seems to become a larger project that back-stimulates the LaTeX filter quite substantially.

[lervag]

I mean the following problem: For each detected mistake, we can provide a certain amount of information. This is not only the explanatory error message itself, but also a list of replacement suggestions from the proofreader (often useful) and the error context as seen in plain text by the proofreader. The latter may be helpful if the LaTeX filter performed some more aggressive modifications, e.g., when handling equations.

All this together may form a quite "long line" for each reported problem, and this may become confusing, even if one uses the error list in the separate quickfix window (:cw). The idea was to make that a bit configurable by auxiliary Vim variables (e.g., g:vimtex_ltyc_show_replacements, g:vimtex_ltyc_show_context), so that the user can extend / restrict the amount of information.

The errorformat syntax is not the problem. I manage to parse oneline and multiline messages from the "linter tool" and can extract the required parts to construct the quickfix messages. But i'm still unsure, whether I completely understand the full potential of the mechanism. It seems to me, that a reconfiguration, depending on the detected error and desired information detail, is only possible by changing the above auxiliary variables (and thus "automatically" the errorformat under the hood) and re-running the compiler. If it is that way, then all is OK. Just do not want to miss a possibility.

Yes, I think you understand the errorformat and quickfix window feature properly.

(My concerns probably have the following root. In all other plugins like ALE, one can display a subwindow that shows all information in a structured manner. Here, we only have a one-liner in the status line, or a wrapped line in the quickfix window.)

No problem. I'll be happy to consider this feature when you are ready!

Thanks again. This seems to become a larger project that back-stimulates the LaTeX filter quite substantially.

Yes, the way you describe it makes me think we sort of want two variants of this feature:

1. A simple, minimal information version that relies on errorformat and the quickfix window. This is useful because it allows a very simple and fast workflow in Vim that follows a very normal standard. I would propose to avoid options for this version, if possible.

2. A more advanced version that must be built either "from scratch" or on top of the quickfix window. This version would allow more options for what to display. If possible, it could also suggest immediate fixups (or a list of choices).

I believe ALE in part (or fully) supports the second variant, including the abillity to apply "fixes" (see :help ale-fix). Perhaps it does not allow an interactive version, I am not sure. I am also not very familiar with writing extensions for ALE, so you would have to investigate that in more detail. But, if it does, then I would propose that you (we?) only implement a simple solution as part of vimtex, and then instead add a more advanced version within the framework of ALE.

[matze-dd]

1. A simple, minimal information version that relies on errorformat and the quickfix window. This is useful because it allows a very simple and fast workflow in Vim that follows a very normal standard. I would propose to avoid options for this version, if possible.

PR #1714 proposes a new vimtex compiler named vlty. I'm unsure about (at least ;-) the following points.

• Is the name vlty OK? It tries to include the component names, but should not be too long.
• Does the configuration via dictionary g:vimtex_vlty fit into vimtex's framework?
• Does the manual section *compiler-vlty* cover all important points? Is it too verbose?
• Does :VimtexReload in the vimrc below have negative side-effects? Is there a "more elegant" way?

Simple example

A vimrc could contain:

set spelllang=en_gb
let g:vimtex_vlty = {'ltdirectory': 'path/to/LanguageTool'}
map <F9> :w<CR>:VimtexReload<CR>:compiler vlty<CR>:make<CR>:cwindow<CR><CR>

Here, the :VimtexReload updates detected document classes and packages. The subsequent :compiler vlty includes these into the command invoked by :make.
(EDIT: Since the LaTeX filter detects \documentclass and \usepackage on its own, this is only necessary for multi-file projects, where a single LaTeX file may lack this information.)

With input

\documentclass{article}                             %  1
\usepackage{amsmath}                                %  2
\usepackage{xcolor}                                 %  3
\begin{document}                                    %  4
\section*{title}                                    %  5
Only few people\footnote{We use \textcolor{color}   %  6
{red color.}}                                       %  7
is lazy.                                            %  8
From~\cite[errx]                                    %  9
{errx}, we have                                     % 10
\begin{equation}                                    % 11
    f(n) = 0 \text{ for for all } n.                % 12
\end{equation}                                      % 13
\end{document}                                      % 14

we get on F9 the quickfix window

|5 col 11|  This sentence does not start with an uppercase letter
|7 col 6|  Possible spelling mistake. 'color' is American English.
|8 col 1|  If 'people' is plural here, don't use the third-person singular verb.
|9 col 12|  Possible spelling mistake found
|12 col 21|  Possible typo: you repeated a word

• Line 5: A typo.
• Line 6: The first argument of \textcolor is skipped ('color').
• Line 7: A typo.
• Line 8: The argument of \footnote has been separated by the LaTeX filter, such that the mistake 'few people is' can be recognised by the proofreader.
• Line 9: The filter translated the citation to [0, errx].
• Line 12: The argument of \text is part of the text seen by the proofreader; see also [here].

EDITS
The default value of g:vimtex_vlty['server'] should be changed from '' to 'no'. (PR updated)
In the documentation, *compiler-vlty* should be replaced by *vimtex-vlty*. (PR updated)
There is a bug in vlty.vim, line 24. (PR updated)

[lervag]

Do you mind if I reply to this comment in #1714 and that we close this issue? It would be easier to have the discussion in a single thread.

[lervag]

I've taken the liberty of doing just this.

[matze-dd]

That's fine for me, too.