Nvim-cmp's keyword_pattern and Vimtex's Omnifunc

[micangl]

Description

Vimtex support in nvim-cmp is an issue which has been extensively discussed. These are just some of the related issues:

• Support for nvim-cmp. #2215
• cmp-omni source not displayed when nvim_lsp is enabled hrsh7th/nvim-cmp#833
• Erorr when using keyword_pattern hrsh7th/nvim-cmp#1170
• Omni Completion (offset?) issues hrsh7th/nvim-cmp#652
• Vimtex support isn't complete hrsh7th/cmp-omni#1
• vimtex support hrsh7th/nvim-cmp#258

The problem explained in those issues dealt mainly with the Vimtex-provided neocomplete pattern (https://github.com/lervag/vimtex/blob/master/autoload/vimtex/re.vim#L35), why it didn't work, and why cite-completion wasn't triggered automatically (i.e. the completion menu appeared) as soon as a \cite{ statement was typed.

I may have found some workarounds and/or hints towards a possible solution. I'll explain them in order:

1. Cite-completion isn't triggered automatically as soon as a \cite{ statement is issued. Differently from other engines, with nvim-cmp the user first has to type a character to have the menu pop up and show bilbiographic references. This is obviously annoying, as the user may not always remember, at the top of his head, exactly which reference to insert (or the key name). A workaround which I don't think was ever discussed is to use a trigger_character in nvim-cmp's configuration, as such:

   { name = "omni", trigger_characters = { "{" } },

2. Correctness of the Vimtex-provided Neocomplete pattern. When the discussions in the previously linked issues were held, the validity of said pattern was questioned and, if I recall correctly, noone ever managed to make it work with nvim-cmp. The pattern, though, is indeed correct. This can be tested by opening the following file in Neovim and sourcing (:so):

   vim.g.regex =
            [=[\%(]=]
         .. [=[\v\\%(]=]
         .. [=[%(\a*cite|Cite)\a*\*?%(\s*\[[^]]*\]){0,2}\s*\{[^}]*]=]
         .. [=[|%(\a*cites|Cites)%(\s*\([^)]*\)){0,2}]=]
         ..    [=[%(%(\s*\[[^]]*\]){0,2}\s*\{[^}]*\})*]=]
         ..    [=[%(\s*\[[^]]*\]){0,2}\s*\{[^}]*]=]
         .. [=[|bibentry\s*\{[^}]*]=]
         .. [=[|%(text|block)cquote\*?%(\s*\[[^]]*\]){0,2}\s*\{[^}]*]=]
         .. [=[|%(for|hy)\w*cquote\*?\{[^}]*}%(\s*\[[^]]*\]){0,2}\s*\{[^}]*]=]
         .. [=[|defbibentryset\{[^}]*}\{[^}]*]=]
         .. [=[|\a*ref%(\s*\{[^}]*|range\s*\{[^,}]*%(}\{)?)]=]
         .. [=[|hyperref\s*\[[^]]*]=]
         .. [=[|includegraphics\*?%(\s*\[[^]]*\]){0,2}\s*\{[^}]*]=]
         .. [=[|%(include%(only)?|input|subfile)\s*\{[^}]*]=]
         .. [=[|([cpdr]?(gls|Gls|GLS)|acr|Acr|ACR)\a*\s*\{[^}]*]=]
         .. [=[|(ac|Ac|AC)\s*\{[^}]*]=]
         .. [=[|includepdf%(\s*\[[^]]*\])?\s*\{[^}]*]=]
         .. [=[|includestandalone%(\s*\[[^]]*\])?\s*\{[^}]*]=]
         .. [=[|%(usepackage|RequirePackage|PassOptionsToPackage)%(\s*\[[^]]*\])?\s*\{[^}]*]=]
         .. [=[|documentclass%(\s*\[[^]]*\])?\s*\{[^}]*]=]
         .. [=[|begin%(\s*\[[^]]*\])?\s*\{[^}]*]=]
         .. [=[|end%(\s*\[[^]]*\])?\s*\{[^}]*]=]
         .. [=[|\a*]=]
         .. [=[)]=]
   local regex_1 = vim.regex(vim.g.regex)
   local match_str = [=[noise \cite{]=]
   
   vim.print(regex_1:match_str(match_str))

3. And now the hardest issue: Making that pattern work with nvim-cmp. I've only partially achieved this goal, and I write this issue with the hope that someone more expert than me will be able to help.

   First thing, nvim-cmp modifies the keyword_pattern provided by the user:
   https://github.com/hrsh7th/nvim-cmp/blob/5dce1b778b85c717f6614e3f4da45e9f19f54435/lua/cmp/context.lua#L77

       return pattern.offset([[\%(]] .. keyword_pattern .. [[\)\m$]], self.cursor_before_line) or self.cursor.col

   This is necessary to ensure that only the last-typed text will be matched, as discussed in Custom completion source - completion popup not showing up hrsh7th/nvim-cmp#1273. The problem is that such a modification assumes to be in magic mode, whilst the keyword pattern we would be using set \v very-magic mode. This can be fixed by either adding a \m at the end of the pattern we pass to nvim-cmp, as such

    .. [=[|\a*]=]
    .. [=[)]=]
    .. [=[\m]=]

   or by correcting nvim-cmp's handling. I still haven't made a PR, but it should be like this:

       return pattern.offset([[\%(]] .. keyword_pattern .. [[\m\)$]], self.cursor_before_line) or self.cursor.col

   After making such an addition, nvim-cmp still doesn't match. It took me multiple hours of debugging but, by modifying cmp/source.lua

   +  local offset = ctx:get_offset(self:get_keyword_pattern()) + 1
   -  local offset = ctx:get_offset(self:get_keyword_pattern())

   nvim-cmp start finally to match correctly and show the completion menu. The issue I've found is that the completion menu, with such a modification, doesn't appear after typing a \cite{ statement. I've tried to troubleshoot this, but I'm not a programmer and the code in that nvim-cmp's function is just too complicated for me.

If someone who has the expertise would like to help, I'd be happy to contribute where I can. I honestly hope that we will, finally, be able to solve this stubborn issue.

Finally, sorry for bearing with me 'till the end of this message.

VimtexInfo

System info:
  OS: Linux 6.4.12-arch1-1
  Vim version: NVIM v0.9.1
  Has clientserver: true
  Servername: /run/user/1000/nvim.896581.0

VimTeX project: a
  base: a.tex
  root: /home/mike
  tex: /home/mike/a.tex
  main parser: current file verified
  document class: article
  compiler: latexmk
    engine: -pdf
    options:
      -verbose
      -file-line-error
      -synctex=1
      -interaction=nonstopmode
    callback: 1
    continuous: 1
    executable: latexmk
  viewer: Zathura
    xwin id: 0
  qf method: LaTeX logfile

[micangl]

@hrsh7th I think some of the points I discussed in this issue could interest you. Feedback from both ends will be beneficial.

[lervag]

@micangl Sorry for not responding sooner! I've been busy with a lot of things, including playing a computer game - which I seldom do. As I don't have much spare time, this means the time for working with VimTeX has been at a minimum for a few weeks.

I'll try to step it up and I'll begin reading your issue post in full now.

[lervag]

I may have found some workarounds and/or hints towards a possible solution. I'll explain them in order:

Thanks; I appreciate your effort and I recognize that you've worked quite hard on this!

Vimtex support in nvim-cmp is an issue …

My initial thought is that I'm using nvim-cmp and I feel like I have very good VimTeX support. The following is my personal configuration (the relevant snippets):

• The start of my lazy.nvim specification of nvim-cmp: https://github.com/lervag/dotnvim/blob/b1dc12bc2d4ecf8f5b5e5c467ccbdd88f5303986/lua/lervag/init/packages.lua#L569
  • My specification is quite lengthy, but most of it is adjustment of formatting and similar and is not so relevant.
• Where I specify my sources for .tex buffers: https://github.com/lervag/dotnvim/blob/b1dc12bc2d4ecf8f5b5e5c467ccbdd88f5303986/lua/lervag/init/packages.lua#L738-L745

For me, this "just works"™. I don't specify any keyword_pattern, and it does not really seem necessary. So, could we first establish that there really is an issue?

Still, I'll give some initial thoughts to your investigations:

1. Cite-completion isn't triggered automatically as soon as a \cite{ statement is issued.

Ah, yes, that's a valid point. Completion triggers for me after typing two letters (due to my settings).

A workaround which I don't think was ever discussed is to use a trigger_character in nvim-cmp's configuration

Cool, thanks! That seems useful!

2. Correctness of the Vimtex-provided Neocomplete pattern. When the discussions in the previously linked issues were held, the validity of said pattern was questioned and, if I recall correctly, noone ever managed to make it work with nvim-cmp. The pattern, though, is indeed correct.

Working with VimTeX has made me quite adept at writing regexes. ;)

3. And now the hardest issue: Making that pattern work with nvim-cmp.

I would ask: do we need that? Could we first establish the current real pain points?

Let's say we recognize some real pain points: I believe a possible better way forward would be to write a full source for nvim-cmp. I believe we could use the existing completion function and possibly the existing regexes for that. I remember that I did attempt this once, but I didnt immediately figure things out and I realized that things worked well enough that I just stopped.

The problem is that such a modification assumes to be in magic mode, whilst the keyword pattern we would be using set \v very-magic mode. This can be fixed by either …

or by adjusting the regex (make a new one) that is magic mode. It shouldn't be too hard.

After making such an addition, nvim-cmp still doesn't match. It took me multiple hours of debugging …

<3

nvim-cmp start finally to match correctly and show the completion menu. …

It would not be surprising to me if such a proposed changed implies other changes that may not be consistent - but I wouldn't know as I've not studied this code.

[micangl]

The main issue, currently, is that one has to add the trigger_characters to make the completion menu appear automatically with \cite{. I've started testing things out by modifying the cmp-omni source.

It would also be very nice if said source would configure formatting for the completion menu, like the solution discussed in issue #2215:

local cmp = require("cmp")
cmp.setup.buffer({
  formatting = {
    format = function(entry, vim_item)
      vim_item.menu = ({
        omni = (vim.inspect(vim_item.menu):gsub('%"', "")),
        luasnip = "[LuaSnip]",
        buffer = "[Buffer]",
      })[entry.source.name]
      return vim_ite
    end,
  },
})

I've tried the pattern with vanilla nvim-cmp and it didn't work, as expected. But it does match without the leading backslashes (which is in accord with that offset issue I've talked about). But as you pointed out, it's just not worth the headache to try making the pattern work with cmp.

The custom source would be nice, though, to set the trigger_characters and formatting. What do you think?

[lervag]

The main issue, currently, is that one has to add the trigger_characters to make the completion menu appear automatically with \cite{.

Ok, but then it also seems as adding the addition config you suggest more or less fixes this, right?

I've started testing things out by modifying the cmp-omni source.

It would also be very nice if said source would configure formatting for the completion menu, like the solution discussed in issue #2215:

Yes, one of the reasons I have a lot of config for nvim-cmp is precisely to improve the formatting of e.g. VimTeX completion with nvim-cmp.

The custom source would be nice, though, to set the trigger_characters and formatting. What do you think?

Well, yes, I still agree. And I would be happy to help figure out these things. Though, I don't have much time. If someone else (you?) would take some initiative, then I could be a "passenger" and would likely be less of a "blocker".

It would be fine by me to consider a PR for this, although I may end up to suggest to add it as an extension for nvim-cmp instead of a part of VimTeX itself.

[micangl]

Yes, what I meant was that we should just create a nvim-cmp extension (plugin). I'm working on it.

[lervag]

Great! Let me know if/when you need input/advice/help!

[micangl]

Well, the hope of creating a custom source for cmp has already faltered. Unfortunately, there's no way for a source to set a format function in a non-destructive way (i.e. without overwriting the user's implementation).

[lervag]

Unfortunately, there's no way for a source to set a format function in a non-destructive way (i.e. without overwriting the user's implementation).

Ah, no, you are probably right. However, I think the conclusion may still be wrong. That is, I don't think we want a source to have a custom formatter, instead, we want it to provide the metadata in a way that is compatible with the nvim-cmp standard format.

That is, the lsp completer looks like this on my end:

I believe we can have a VimTeX completer for nvim-cmp that translates information for the completion items from the omnicomplete function into a data model that fits with nvim-cmp.

[micangl]

The custom format function which I planned to implement simply removed double quotes from the completion items. Adding it, gets you this result (no quotes surrounding the book titles:

I believe we can have a VimTeX completer for nvim-cmp that translates information for the completion items from the omnicomplete function into a data model that fits with nvim-cmp.

What would you like to achieve, exactly?

[lervag]

On my end, without changing the formatter option, I get a completion menu that looks like this:

I've made some adjustments which basically just changes Text to a symbol (but it does more for other completion types):

What would you like to achieve, exactly?

I'm not very familiar with the spec for nvim-cmp, but I believe it could be possible to leverage it to get a better experience. E.g., we could move the full info into a preview window.

The complete function should call a callback function with one or more completion items according to this spec:

https://github.com/hrsh7th/nvim-cmp/blob/5dce1b778b85c717f6614e3f4da45e9f19f54435/lua/cmp/types/lsp.lua#L264-L284

There may be some better docs for this that I have not found yet, but I think this may point to a direction of what I want to achieve.

[micangl]

On my end, without changing the formatter option, I get a completion menu that looks like this:

The citations section of the menu looks strange; mine, even without the format function, doesn't present such an issue. Might it be a problem with the .bib file?

I've added an option to display the full Info in the documentation window:

I think that the result is quite lame, especially considering the huge amount a .bib file may provide:

@article{knuth:1984,
  title={Literate Programming},
  author={Donald E. Knuth},
  journal={The Computer Journal},
  volume={27},
  number={2},
  pages={97--111},
  year={1984},
  publisher={Oxford University Press}
}

I've noticed that the info_fmt variable can only accept a small set of parameters; maybe we should add support for the other Bibtex entries? Having all of the bibliographic data in the documentation window would really be game-changing, I think.

I've made some adjustments which basically just changes Text to a symbol (but it does more for other completion types):

Should I add an option to set this too?

Another game-changing option is the ability to fuzzy match, not only against the citation keywords, but also against the data contained in the menu field (like author, publication date, and title):

P.S. I still haven't uploaded the repository on Github, but I will soon (I need to make the codebase presentable).