biber increases compile time and creates latency for big documents

[pchest]

Description

I am a neovim user that regularly uses vimtex to edit and compile latex documents. Recently I've been working on a document that is 79 pages with figures and references. I found that when I change the backend for autocite to biber from bibtex that I encountered the following issue: After I initially compile the document, edits create extreme latency that renders the cursor unmovable. The full time it takes from the edit and render to being able to move the cursor again is 30 seconds on average. When I switched the backend from biber to bibtex, the render time decreased to 9 seconds on average and the latency issue disappeared entirely.

I have encountered this issue on multiple devices running pop os 22.04. The neovim environment I am using is 10.4 with pchest/nvim-config as the configuration setup.

Steps to reproduce

No response

Expected behavior

No response

Actual behavior

No response

Do you use a latexmkrc file?

No

VimtexInfo

I am using neovim, so I don't have a VimtexInfo page. Here is the result of :checkhealth:

==============================================================================
coc: health#coc#check

- OK nvim version satisfied
- OK Environment check passed
- OK Javascript bundle build/index.js found
- OK Service started

==============================================================================
codecompanion: require("codecompanion.health").check()

- Neovim version: 0.10.4
- Log file: /home/patrick/.local/state/nvim/codecompanion.log

Dependencies: ~
- OK plenary.nvim installed
- OK nvim-treesitter installed

Tree-sitter parsers: ~
- OK markdown parser installed
- OK yaml parser installed

Libraries: ~
- OK curl installed

==============================================================================
diffview: require("diffview.health").check()

Checking plugin dependencies ~
- OK nvim-web-devicons installed.

Checking VCS tools ~
- The plugin requires at least one of the supported VCS tools to be valid.
- OK Git found.
- OK Git is up-to-date. (2.34.1)
- WARNING Configured `hg_cmd` is not executable: 'hg'

==============================================================================
dressing: require("dressing.health").check()

dressing.nvim ~
- OK vim.ui.input active
- OK vim.ui.select active: telescope

==============================================================================
fzf_lua: health#fzf_lua#check

fzf-lua [required] ~
- OK 'fzf' `0.29 (devel)`
- OK 'git' `git version 2.34.1`
- OK 'rg' `ripgrep 13.0.0`
- OK 'fdfind' `fd 8.3.1`
- WARNING 'fzf' `>= 0.53` is recommended.

fzf-lua [optional] ~
- OK `nvim-web-devicons` found
- OK 'rg' `ripgrep 13.0.0`
- OK 'fdfind' `fd 8.3.1`

fzf-lua [optional:media] ~
- WARNING 'viu' not found
- WARNING 'chafa' not found
- WARNING 'ueberzugpp' not found

fzf-lua [env] ~
- OK `FZF_DEFAULT_OPTS` is not set
- OK `FZF_DEFAULT_OPTS_FILE` is not set

==============================================================================
hop: require("hop.health").check()

Ensuring keys are unique ~
- OK Keys are unique

Checking for deprecated features ~
- OK All good

==============================================================================
lazy: require("lazy.health").check()

lazy.nvim ~
- {lazy.nvim} version `11.16.2`
- OK {git} `version 2.34.1`
- WARNING found existing packages at `/home/patrick/.local/share/nvim/site/pack/packer`
- OK packer_compiled.lua not found

luarocks ~
- OK luarocks disabled

==============================================================================
lspconfig: require("lspconfig.health").check()

LSP configs active in this session (globally) ~
- Configured servers: pylsp, bashls, vimls, ltex
- OK Deprecated servers: (none)

LSP configs active in this buffer (bufnr: 1) ~
- Language client log: ~/.local/state/nvim/lsp.log
- Detected filetype: `dashboard`
- 0 client(s) attached to this buffer

Docs for active configs: ~

==============================================================================
molten: require("molten.health").check()

molten-nvim ~
- OK NeoVim >=0.9
- OK Python >=3.10
- OK Python module pynvim found
- OK Python module jupyter-client found
- OK Python module cairosvg found
- OK Python module pnglatex found
- OK Python module plotly found
- OK Python module kaleido found
- OK Python module pyperclip found
- OK Python module nbformat found
- OK Python module pillow found

==============================================================================
nvim: require("nvim.health").check()

Configuration ~
- OK no issues found

Runtime ~
- OK $VIMRUNTIME: /home/linuxbrew/.linuxbrew/Cellar/neovim/0.10.4/share/nvim/runtime

Performance ~
- OK Build type: Release

Remote Plugins ~
- OK Up to date

terminal ~
- key_backspace (kbs) terminfo entry: `key_backspace=^H`
- key_dc (kdch1) terminfo entry: `key_dc=\E[3~`
- $COLORTERM="truecolor"

External Tools ~
- OK ripgrep 13.0.0 (/usr/bin/rg)

==============================================================================
nvim-treesitter: require("nvim-treesitter.health").check()

Installation ~
- OK `tree-sitter` found 0.24.7 (parser generator, only needed for :TSInstallFromGrammar)
- OK `node` found v19.3.0 (only needed for :TSInstallFromGrammar)
- OK `git` executable found.
- OK `cc` executable found. Selected from { vim.NIL, "cc", "gcc", "clang", "cl", "zig" }
  Version: cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
- OK Neovim was compiled with tree-sitter runtime ABI version 14 (required >=13). Parsers must be compatible with runtime ABI.

OS Info:
{
  machine = "x86_64",
  release = "6.12.10-76061203-generic",
  sysname = "Linux",
  version = "#202412060638~1740154617~22.04~b4b3ebc SMP PREEMPT_DYNAMIC Fri F"
} ~

Parser/Features         H L F I J
  - c                   ✓ ✓ ✓ ✓ ✓
  - lua                 ✓ ✓ ✓ ✓ ✓
  - luadoc              ✓ . . . .
  - markdown            ✓ . ✓ ✓ ✓
  - markdown_inline     ✓ . . . ✓
  - python              ✓ ✓ ✓ ✓ ✓
  - query               x ✓ ✓ ✓ ✓
  - r                   ✓ ✓ . ✓ ✓
  - rnoweb              ✓ . ✓ . ✓
  - vim                 ✓ ✓ ✓ . ✓
  - vimdoc              ✓ . . . ✓
  - yaml                ✓ ✓ ✓ ✓ ✓

  Legend: H[ighlight], L[ocals], F[olds], I[ndents], In[j]ections
         +) multiple parsers found, only one will be used
         x) errors found in the query, try to run :TSUpdate {lang} ~

The following errors have been detected: ~
- ERROR query(highlights): ...m/0.10.4/share/nvim/runtime/lua/vim/treesitter/query.lua:252: Query error at 14:2. Invalid node type "missing_node":
  (missing_node
   ^
  
  query(highlights) is concatenated from the following files:
  | [ERROR]:"/home/patrick/.local/share/nvim/lazy/nvim-treesitter/queries/query/highlights.scm", failed to load: ...m/0.10.4/share/nvim/runtime/lua/vim/treesitter/query.lua:252: Query error at 14:2. Invalid node type "missing_node":
  (missing_node
   ^
  

==============================================================================
provider.clipboard: require("provider.clipboard.health").check()

Clipboard (optional) ~
- OK Clipboard tool found: xclip

==============================================================================
provider.node: require("provider.node.health").check()

Node.js provider (optional) ~
- Disabled (loaded_node_provider=0).

==============================================================================
provider.perl: require("provider.perl.health").check()

Perl provider (optional) ~
- Disabled (loaded_perl_provider=0).

==============================================================================
provider.python: require("provider.python.health").check()

Python 3 provider (optional) ~
- Using: g:python3_host_prog = "/home/patrick/anaconda3/bin/python3"
- Executable: /home/patrick/anaconda3/bin/python3
- Python version: 3.12.2
- pynvim version: 0.5.0
- OK Latest pynvim is installed.

Python virtualenv ~
- OK no $VIRTUAL_ENV

==============================================================================
provider.ruby: require("provider.ruby.health").check()

Ruby provider (optional) ~
- Disabled (loaded_ruby_provider=0).

==============================================================================
r: require("r.health").check()

Checking applications and plugins: ~
- OK Neovim version: 0.10.4
- OK C compiler (`gcc` or `clang`) found.
- OK `R-nvim/cmp-r` found.
- OK `nvim-treesitter/nvim-treesitter` found.

Checking tree-sitter parsers: ~
- OK `r`  found.
- OK `markdown`  found.
- OK `markdown_inline`  found.
- OK `rnoweb`  found.
- ERROR `latex` not found.
- OK `yaml`  found.

==============================================================================
render-markdown: require("render-markdown.health").check()

render-markdown.nvim [version] ~
- OK plugin 8.0.3
- OK neovim >= 0.10

render-markdown.nvim [configuration] ~
- OK valid

render-markdown.nvim [treesitter] ~
- OK markdown: parser installed
- OK markdown_inline: parser installed
- WARNING latex: parser not installed
  - ADVICE:
    - Disable LaTeX support to avoid this warning by setting { latex = { enabled = false } }
- OK markdown: highlight enabled

render-markdown.nvim [icons] ~
- OK using: nvim-web-devicons

render-markdown.nvim [executables] ~
- WARNING latex2text: not installed
  - ADVICE:
    - Disable LaTeX support to avoid this warning by setting { latex = { enabled = false } }

render-markdown.nvim [conflicts] ~
- OK headlines: not installed
- OK obsidian: not installed

==============================================================================
targets: health#targets#check

- WARNING Conflicting mapping found:
  iB → :<C-U>call text_obj#Buffer()<CR>
  B → {'pair': [{'c': '}', 'o': '{'}]}
- WARNING Conflicting mapping found:
  ab → <Plug>(textobj-sandwich-auto-a)
  b → {'pair': [{'c': ')', 'o': '('}, {'c': ']', 'o': '['}, {'c': '}', 'o': '{'}]}
- WARNING Conflicting mapping found:
  ib → <Plug>(textobj-sandwich-auto-i)
  b → {'pair': [{'c': ')', 'o': '('}, {'c': ']', 'o': '['}, {'c': '}', 'o': '{'}]}

==============================================================================
telescope: require("telescope.health").check()

Checking for required plugins ~
- OK plenary installed.
- OK nvim-treesitter installed.

Checking external dependencies ~
- OK rg: found ripgrep 13.0.0
- OK fd: found fd 8.3.1

===== Installed extensions ===== ~

Telescope Extension: `notify` ~
- No healthcheck provided

==============================================================================
vim.lsp: require("vim.lsp.health").check()

- LSP log level : WARN
- Log path: /home/patrick/.local/state/nvim/lsp.log
- WARNING Log size: 265427 KB

vim.lsp: Active Clients ~
- No active clients

vim.lsp: File Watcher ~
- file watching "(workspace/didChangeWatchedFiles)" disabled on all clients

vim.lsp: Position Encodings ~
- No active clients

==============================================================================
vim.treesitter: require("vim.treesitter.health").check()

- Nvim runtime ABI version: 14
- OK Parser: lua                  ABI: 14, path: /home/patrick/.local/share/nvim/lazy/nvim-treesitter/parser/lua.so
- OK Parser: luadoc               ABI: 14, path: /home/patrick/.local/share/nvim/lazy/nvim-treesitter/parser/luadoc.so
- OK Parser: markdown             ABI: 14, path: /home/patrick/.local/share/nvim/lazy/nvim-treesitter/parser/markdown.so
- OK Parser: markdown_inline      ABI: 14, path: /home/patrick/.local/share/nvim/lazy/nvim-treesitter/parser/markdown_inline.so
- OK Parser: python               ABI: 14, path: /home/patrick/.local/share/nvim/lazy/nvim-treesitter/parser/python.so
- OK Parser: r                    ABI: 14, path: /home/patrick/.local/share/nvim/lazy/nvim-treesitter/parser/r.so
- OK Parser: rnoweb               ABI: 13, path: /home/patrick/.local/share/nvim/lazy/nvim-treesitter/parser/rnoweb.so
- OK Parser: vim                  ABI: 14, path: /home/patrick/.local/share/nvim/lazy/nvim-treesitter/parser/vim.so
- OK Parser: vimdoc               ABI: 14, path: /home/patrick/.local/share/nvim/lazy/nvim-treesitter/parser/vimdoc.so
- OK Parser: yaml                 ABI: 14, path: /home/patrick/.local/share/nvim/lazy/nvim-treesitter/parser/yaml.so
- OK Parser: c                    ABI: 14, path: /home/linuxbrew/.linuxbrew/Cellar/neovim/0.10.4/lib/nvim/parser/c.so
- OK Parser: lua                  ABI: 14, path: /home/linuxbrew/.linuxbrew/Cellar/neovim/0.10.4/lib/nvim/parser/lua.so
- OK Parser: markdown             ABI: 14, path: /home/linuxbrew/.linuxbrew/Cellar/neovim/0.10.4/lib/nvim/parser/markdown.so
- OK Parser: markdown_inline      ABI: 14, path: /home/linuxbrew/.linuxbrew/Cellar/neovim/0.10.4/lib/nvim/parser/markdown_inline.so
- OK Parser: query                ABI: 14, path: /home/linuxbrew/.linuxbrew/Cellar/neovim/0.10.4/lib/nvim/parser/query.so
- OK Parser: vim                  ABI: 14, path: /home/linuxbrew/.linuxbrew/Cellar/neovim/0.10.4/lib/nvim/parser/vim.so
- OK Parser: vimdoc               ABI: 14, path: /home/linuxbrew/.linuxbrew/Cellar/neovim/0.10.4/lib/nvim/parser/vimdoc.so

==============================================================================
which-key: require("which-key.health").check()

- OK Most of these checks are for informational purposes only.
  WARNINGS should be treated as a warning, and don't necessarily indicate a problem with your config.
  Please |DON'T| report these warnings as an issue.

Checking your config ~
- WARNING |mini.icons| is not installed
- OK |nvim-web-devicons| is installed

Checking for issues with your mappings ~
- OK No issues reported

checking for overlapping keymaps ~
- WARNING In mode `n`, <c> overlaps with <cc>:
  - <c>: Change
- WARNING In mode `n`, <gc> overlaps with <gcc>, <gcu>:
  
- WARNING In mode `n`, <,<Space>> overlaps with <,<Space>f<Space>f>, <,<Space>f<Space>g>, <,<Space>f<Space>r>, <,<Space>e<Space>v>:
  - <,<Space>>: remove trailing space
  - <,<Space>f<Space>f>: Dashboard-action: Find  File                              
  - <,<Space>f<Space>g>: Dashboard-action: Project grep                            
  - <,<Space>f<Space>r>: Dashboard-action: Recently opened files                   
  - <,<Space>e<Space>v>: Dashboard-action: Open Nvim config                        
- WARNING In mode `n`, <sr> overlaps with <srb>:
  
- WARNING In mode `n`, <sd> overlaps with <sdb>:
  
- WARNING In mode `i`, <,> overlaps with <,r;>:
  - <,r;>: insert R pipe
- WARNING In mode `x`, <@> overlaps with <@(targets)>:
  - <@>: :help |v_@-default|
- WARNING In mode `x`, <a> overlaps with <aI>, <as>, <ab>, <ai>:
  - <a>: around
  - <as>: sentence
  - <ab>: [(]) block
- WARNING In mode `x`, <i> overlaps with <iI>, <iu>, <ii>, <iB>, <is>, <ib>:
  - <i>: inside
  - <iu>: URL text object
  - <iB>: buffer text object
  - <is>: inner sentence
  - <ib>: inner [(])
- WARNING In mode `o`, <a> overlaps with <aI>, <as>, <ab>, <ai>:
  - <a>: around
  - <as>: sentence
  - <ab>: [(]) block
- WARNING In mode `o`, <i> overlaps with <iI>, <iu>, <ii>, <iB>, <is>, <ib>:
  - <i>: inside
  - <iu>: URL text object
  - <iB>: buffer text object
  - <is>: inner sentence
  - <ib>: inner [(])
- OK Overlapping keymaps are only reported for informational purposes.
  This doesn't necessarily mean there is a problem with your config.

Checking for duplicate mappings ~
- OK No duplicate mappings found

[lervag]

I am using neovim, so I don't have a VimtexInfo page. Here is the result of :checkhealth:

No, sorry, that's wrong. I found, by looking at your config, that you are lazy loading VimTeX. I've added a warning about this both in the README and in the docs - so please, stop that!

I.e., remove this line:

    ft = { "tex" },

I would generally advice that you read (or at least skim) the README file and the introduction in the VimTeX docs. There's a lot of useful info there.

I have encountered this issue on multiple devices running pop os 22.04. The neovim environment I am using is 10.4 with pchest/nvim-config as the configuration setup.

Can you see if you can reproduce your issue with a minimal configuration, e.g. something like this:

set nocompatible
" CHECK IF THE PATH HERE IS RIGHT!
set runtimepath^=~/.local/share/nvim/lazy/lazy.nvim/vimtex
set runtimepath+=~/.local/share/nvim/lazy/lazy.nvim/vimtex/after
filetype plugin indent on
syntax enable

Save it to test.vim in the directory where you encounter the problem, then run nvim --clean -u test.vim FILE.tex where FILE.tex is the document that does not work as expected. Use \ll to start compiling and then check if the problem persists.

[pchest]

I reproduced the lag effect with the minimal setup you provided and a document with a biber backend. Removing lazy load doesn't seem to affect this issue in my original setup either. When I make a change to the .tex document, the lag kicks in immediately after the pdf updates.

[lervag]

I reproduced the lag effect with the minimal setup you provided and a document with a biber backend. ... When I make a change to the .tex document, the lag kicks in immediately after the pdf updates.

Ok, then it's strange that I can't reproduce also. Did I understand correctly that the following steps should reproduce:

1. Download the arXiv source manuscript.
2. Use the simple minimal test.vim file I provided above.
3. Run nvim --clean -u test.vim FILE.tex.
4. Start compilation \ll.
5. Close (or just ignore) the viewer.
6. Make a change to the document and save. Immediately after press and hold e.g. j or k og <c-u> or <c-d> or similar.

After 6., you say you observe a very clear lag. Right?

Removing lazy load doesn't seem to affect this issue in my original setup either.

And it shouldn't. Lazy loading is not relevant to this issue, I just noticed that you used lazy loading and adviced that you should not do that.

[lervag]

By the way, it would be nice if you ran :VimtexInfo and reported the output.

[pchest]

I completed steps 2-6, but for step 1 I'm using my own manuscript which is about 80 pages in length and includes Chinese characters. Is there a particular document you'd like me to test?

Also, here's the output of :VimtexInfo for the minimal setup you provided:

System info:
  OS: Pop!_OS 22.04 LTS
  Vim version: NVIM v0.10.4
  Has clientserver: true
  Servername: /run/user/1000/nvim.30843.0

VimTeX project: main
  base: main.tex
  root: /home/patrick/Sync/Chinese_Foreign_Coverage/05_paper/PSRM
  tex: /home/patrick/Sync/Chinese_Foreign_Coverage/05_paper/PSRM/main.tex
  main parser: current file verified
  document class: article
  packages: CJKfntef adjcalc adjustbox algorithm2e amsbsy amsgen amsmath amsopn amstext appendix array atbegshi atbegshi-ltx atveryend atveryend-ltx authblk auxhook babel biblatex biblatex-chicago bigintcalc bigstrut bitset blx-case-expl3 booktabs calc caption caption3 collectbox color csquotes ctex ctexhook ctexpatch dcolumn etexcmds etoolbox expl3 fancyhdr fix-cm float fontenc fontspec fontspec-xetex footmisc geometry gettitlestring graphics graphicx hycolor hyperref ifoddpage iftex ifthen ifvtex indentfirst infwarerr intcalc keyval kvdefinekeys kvoptions kvsetkeys l3keys2e letltxmacro logreq lscape ltxcmds multirow nameref pdfescape pdflscape pdftexcmds pgf pgfcomp-version-0-65 pgfcomp-version-1-18 pgfcore pgffor pgfkeys pgfmath pgfrcs pgfsys refcount relsize rerunfilecheck rotating setspace siunitx stringenc threeparttable tikz todonotes translations trig trimclip underscore uniquecounter url varwidth xcolor xeCJK xeCJKfntef xkeyval xparse xspace xstring xtemplate zhnumber
  source files:
    main.tex
    reg_table1_short.tex
    reg_table_h34_short.tex
    reg_rob_h12_sent.tex
    tab_sample.tex
    reg_table1.tex
    control.tex
    reg_rob_h34_placebo.tex
    reg_table2.tex
    reg_rob_h12_dictionary.tex
    reg_rob_h34_dictionary.tex
    reg_rob_h12_win.tex
    reg_rob_h34_win.tex
    reg_rob_h34_art.tex
    reg_rob_h12_altdv.tex
    reg_rob_h34_altdv.tex
    reg_rob_h12_alt_date.tex
  compiler: latexmk
    engine: -xelatex
    options:
      -verbose
      -file-line-error
      -synctex=1
      -interaction=nonstopmode
    callback: 1
    continuous: 1
    executable: latexmk
  viewer: General
  qf method: LaTeX logfile

[lervag]

I completed steps 2-6, but for step 1 I'm using my own manuscript which is about 80 pages in length and includes Chinese characters. Is there a particular document you'd like me to test?

Sorry, I mixed this issue with another. I want to reproduce your problem and it would help a lot if you could share a document with which I can reproduce it. Perhaps you could make a reduced and simplified version of what you are working on without any "relevant" content?

Or you could find some example document and adjust it so that it will reproduce the problem?

[pchest]

OK, I made a sample that replicates the problem. Should I send it to your Gmail address?

[lervag]

You can either send it to my email address or attach it as a file to this issue thread. Both are fine for me.

[lervag]

Context (for future self and possibly others): I've received an example document by email.

I can reproduce the issue now. It seems to me this is not related to the backend, but to the use of biblatex. I've found that the lag is due to the following function:

vimtex/autoload/vimtex/qf/biblatex.vim

Lines 133 to 145 in ffd09d8

	function! s:qf.get_key_lnum(key, filename) abort dict " {{{1
	if !filereadable(a:filename) | return 0 | endif
	let l:lines = readfile(a:filename)
	let l:lnums = range(len(l:lines))
	let l:annotated_lines = map(l:lnums, '[v:val, l:lines[v:val]]')
	let l:matches = filter(l:annotated_lines,
	\ {_, x -> x[1] =~# '^\s*@\w*{\s*\V' . a:key})
	return len(l:matches) > 0 ? l:matches[-1][0]+1 : 0
	endfunction
	" }}}1

Now, I notice you are using an extremely large bibliography file. I can see how that feels convenient, but I strongly recommend that you avoid that. In this case, just reading the file takes more than a second in Vimscript.

Notice that there are relatively good tools to simplify this. For instance, you can use bibtool for this. Something like this: bibtool -x file.aux -o cited.bib -i bibliography.bib. This did not work on my end, but that might be because I was not able to properly compile your document (I miss a few packages - and when I tried to remove them by commenting them out, I got

Now, we could look into finding a way to cope with this issue in VimTeX as well. The point of the slow function here is to help pinpoint errors with bib entries. So one idea that pops to mind is to avoid the additional postprocessing for large bib files - the drawback is less useful quickfix entries.

[lervag]

Ok, for some reason I was not able to extract the bib entries with bibtool even when the document compiled properly. Perhaps it does not fully support biblatex or something like that, I don't know.

However, I found the following suggestion: https://tex.stackexchange.com/a/261045/34697. I.e., to extract just the entries you are using you can use biber itself. For the example you sent me: biber main_small --output_format bibtex. It will produce main_small_biber.bib that contains just the entries that you used in the document.

[lervag]

I've pushed a small change that will avoid the expensive post processing if the bib file is very large (larger than 200kB). I believe this may resolve your issue. Let me know if it works.

Now, I still strongly recommend that you avoid a huge shared bib file. :)

[pchest]

Wow, thanks for looking into this issue! I admit that I did export far more articles than I needed into my bib file. I had issues applying the bibtool package as well, but the biber command worked like a charm!

So one idea that pops to mind is to avoid the additional postprocessing for large bib files - the drawback is less useful quickfix entries.

I think skipping post-processing for large bib files makes sense. You could specify a bib length threshold after which it kicks in, along with a warning that tells users their bib file is too long to implement checks for bibtex errors. The trick then would be identifying an appropriate threshold number.