pplatex something went wrong with parsing log files

[BertrandSim]

Description

I recently switched to using pplatex to parse log files (Previously I used latexlog, but in a multifile project, the quickfix list showed that the errors/warnings were in the main file, when it should be in the subfiles).

Now, when I hit \le, I get the following error: "Something went wrong when parsing log files!"

Here is the relevant log file: orient_metric.log

Running pplatex -i orient_metric.log, I get the following output

** Warning in orient_metric.tex:
   so3_matrix_norm' on page 1 undefined on input line 43.

** Warning in orient_metric.tex:
    Reference `section_model_main' on page 1 undefined on input line177.

** Warning in orient_metric.tex:
   orient_samp' on page 1 undefined on input line 181.

** Warning in orient_metric.tex:  There were undefined references.

Result: o) Errors: 0, Warnings: 4, BadBoxes: 0

The relevant vimtex config are as follows:

let g:vimtex_view_general_viewer = 'SumatraPDF'
" setup forward and backward search
let g:vimtex_view_general_options
	  \ = '-reuse-instance -forward-search @tex @line @pdf'
	  \ . ' -inverse-search "gvim --servername ' . v:servername
	  \ . ' --remote-send \"^<C-\^>^<C-n^>'
	  \ . ':drop \%f^<CR^>:\%l^<CR^>:normal\! zzzv^<CR^>'
	  \ . ':execute ''drop '' . fnameescape(''\%f'')^<CR^>'
	  \ . ':\%l^<CR^>:normal\! zzzv^<CR^>'
	  \ . ':call remote_foreground('''.v:servername.''')^<CR^>^<CR^>\""'
let g:vimtex_view_general_options_latexmk = '-reuse-instance'
" open in a new tab for a backward search
let g:vimtex_view_reverse_search_edit_cmd = 'tabedit'

" use single shot compilation instead of continuous mode
let g:vimtex_compiler_latexmk = {'continuous': 0}

" use pplatex to preprocess latexlog output
let g:vimtex_quickfix_method = 'pplatex'

Let me know if more info is required!

Steps to reproduce

No response

Expected behavior

No response

Actual behavior

No response

Do you use a latexmkrc file?

No

VimtexInfo

System info
  OS: Microsoft Windows 10 Pro (10.0.19042 N/A Build 19042)
  Vim version: VIM 8.1 (1-2424)
  Has clientserver: true
  Servername: GVIM1

VimTeX project: orient_metric
  base: orient_metric.tex
  root: C:/Users/Bertrand/Documents/Bert/Uni Postgrad/Research/Thesis
  tex: C:/Users/Bertrand/Documents/Bert/Uni Postgrad/Research/Thesis/orient_metric.tex
  out: C:/Users/Bertrand/Documents/Bert/Uni Postgrad/Research/Thesis/orient_metric.pdf
  log: C:/Users/Bertrand/Documents/Bert/Uni Postgrad/Research/Thesis/orient_metric.log
  aux: C:/Users/Bertrand/Documents/Bert/Uni Postgrad/Research/Thesis/orient_metric.aux
  fls: C:/Users/Bertrand/Documents/Bert/Uni Postgrad/Research/Thesis/orient_metric.fls
  main parser: local file
  compiler: latexmk
    configuration: 
      continuous: 0
      callback: 1
      latexmk options:
        -verbose
        -synctex=1
        -interaction=nonstopmode
      latexmk engine: -pdf
    cmd: set max_print_line=2000 & latexmk -verbose -synctex=1 -interaction=nonstopmode -pdf "orient_metric.tex"
  viewer: General
    process: 
      pid: -
      cmd: start /b cmd /s /c "SumatraPDF -reuse-instance -forward-search "C:/Users/Bertrand/Documents/Bert/Uni Postgrad/Research/Thesis/orient_metric.tex" 210 "C:\Users\Bertrand\Documents\Bert\Uni Postgrad\Research\Thesis\orient_metric.pdf" -inverse-search "gvim --servername GVIM1 --remote-send \"^<C-\^>^<C-n^>:drop \%f^<CR^>:\%l^<CR^>:normal\! zzzv^<CR^>:execute 'drop ' . fnameescape('\%f')^<CR^>:\%l^<CR^>:normal\! zzzv^<CR^>:call remote_foreground('GVIM1')^<CR^>^<CR^>\"" >nul"
  qf: LaTeX logfile using pplatex
    addqflist: 330
    set_errorformat: 329
  document class: subfiles
  packages:
    amsbsy
    amsfonts
    amsgen
    amsmath
    amsopn
    amssymb
    amstext
    amsthm
    atbegshi
    atbegshi-ltx
    atveryend
    atveryend-ltx
    auxhook
    bbm
    bigintcalc
    bitset
    calc
    caption
    caption3
    cleveref
    color
    epstopdf-base
    etexcmds
    expl3
    gettitlestring
    graphics
    graphicx
    hycolor
    hyperref
    iftex
    import
    infwarerr
    intcalc
    keyval
    kvdefinekeys
    kvoptions
    kvsetkeys
    letltxmacro
    ltxcmds
    mathtools
    mhsetup
    nameref
    natbib
    pdfescape
    pdftexcmds
    pgf
    pgfcomp-version-0-65
    pgfcomp-version-1-18
    pgfcore
    pgffor
    pgfkeys
    pgfmath
    pgfrcs
    pgfsys
    physics
    refcount
    rerunfilecheck
    subcaption
    subfiles
    tikz
    trig
    uniquecounter
    url
    xcolor
    xparse

[lervag]

I believe this should work now, please test.

[BertrandSim]

Sorry, but I still get the same error: "Something went wrong...". I honestly have no idea how to further troubleshoot the problem. (A minimal main.tex and subfile sub.tex works fine, with the quickfix list shown)

[lervag]

Ok. Well, I can't really help much if you can't describe how to reproduce the problem reliably... :(

One possibility is this: In a project where you have this problem, do this:

1. First ensure you can reproduce the problem with your normal config and the specific project.
2. Copy the .blg file and the .log file into a new directory and create a minimal .tex file.
3. Open the new .tex file with Vim with normal config, check if \le still has the same problem.
4. If yes: Now use a minimal vimrc file and check again.
5. Now check if removing either the .blg or the .log file resolves the issue.

Now, if we're lucky, this means that it is sufficient to share either the .log file or the .blg file from your project for me to reproduce the problem.

[BertrandSim]

Hi lervag, apologies for my late reply.

Appreciate your response, I checked again, but unfortunately, I could not resolve it. The details are as below.

This problem happens for my usual config, and for this specific project. Each time I press \le for this specific project and config, I get the "Something went wrong when parsing log files" error.

I followed the steps you outlined above:

1. First ensure you can reproduce the problem with your normal config and the specific project.

Yes, reproducible.

2. Copy the .blg file and the .log file into a new directory and create a minimal .tex file.

Ok.

3. Open the new .tex file with Vim with normal config, check if \le still has the same problem.

There is now no problem here. The quickfix window opens, with the following contents:

|45 warning| so3_matrix_norm' on page 1 undefined
|177 warning|  Reference `section_model_main' on page 1 undefined
|181 warning| orient_samp' on page 1 undefined

So, it seems that the error is only for this particular tex file (or actually, this particular project across multiple files). I cleared all aux and intermediate files, leaving only the tex files, and compiled with \ll again. However, on \le, the error still persists.

Do you have any impression as to where the problem might be?

[lervag]

Just to be pedantic: After you have done step 1, can you repeat it and make sure to only do \le? (This may already be exactly what you are doing, but I just want to be sure.)

Next, in step 2: are you using the same file names as the original files? If not, please try that and report.

If that did not bring us closer, then we need to dig deeper. Can you go to qf.vim and comment out these lines:

vimtex/autoload/vimtex/qf.vim

Lines 55 to 60 in f1db6a5

	catch
	call vimtex#log#error('Something went wrong when parsing log files!')
	if g:vimtex_quickfix_mode > 0
	cclose
	endif
	return

The idea is to allow the full error message to be shown. That might help locate the origin of the problem.

Another possibility is for you to slowly simplify your project until the problem disappears. E.g. comment out half of your content; if it still fails, continue with the next half. And so on, sort of like a bisection. With this method, it should not take too long to find the lines that result in the error message.

Yet another possibility is for you to share the original source files with me. I understand that this may not be desirable, so that's only left as a suggestion in case you're comfortable with it.

[BertrandSim]

Looks like the root cause has been found --- spaces in filenames, or filepaths.

2. Copy the .blg file and the .log file into a new directory and create a minimal .tex file.

in step 2: are you using the same file names as the original files? If not, please try that and report.

Yes, same file names were used. However, I noticed that the directory of my actual project has a space in it.
In my test project, the directory had no space.

To illustrate, the full path to the tex file in my test project is
"C:/Users/Bertrand/Desktop/a1/orient_metric.tex". \le works as intended here.
If I change "a1" to something with a space, such as
"C:/Users/Bertrand/Desktop/a1 a2/orient_metric.tex", pressing \le produces the error.
Similarly, if the filename has a space such as
"C:/Users/Bertrand/Desktop/a1/orient metric.tex", pressing \le produces the same error.

Can you go to qf.vim and comment out these lines

Commenting lines 55-60 in vimtex/autoload/vimtex/qf.vim gives the error message:

"~\Desktop\a1\orient metric.tex" [unix] 214L, 10062C
Error detected while processing function vimtex#qf#toggle[4]..vimtex#qf#open[4]..vimtex#qf#setqflist[25]..334[9]..vimtex#qf#u#caddfile:
line    7:
E40: Can't open errorfile C:\Users\Bertrand\Desktop\a1/orient metric.pplatex
Error detected while processing function vimtex#qf#toggle:
line    4:
E171: Missing :endif

Finally, I'm on a windows OS (have not tested on Mac or Linux).

[lervag]

Ah, nice, then we're getting closer. I still can't reproduce this on Linux, though, so it seems there is something that may be problematic only on Windows.

Could you add some unsilent echos as specified below and report what they show when you have spaces in the paths?

https://github.com/lervag/vimtex/blob/master/autoload/vimtex/qf/pplatex.vim#L82-L93

  unsilent echo 'DBG1' printf('pplatex -i %s >%s', l:log, l:tmp)
  silent call system(printf('pplatex -i %s >%s', l:log, l:tmp))
  unsilent echo 'DBG2' l:tmp
  call vimtex#qf#u#caddfile(self, l:tmp)
  silent call delete(l:tmp)

vimtex/autoload/vimtex/qf/u.vim

Lines 7 to 16 in f1db6a5

	function! vimtex#qf#u#caddfile(qf, file) abort " {{{1
	" This is a utility function for loading errors from a file into the quickfix
	" window with ":caddfile" without calling possibly defined QuickFixCmdPost
	" autotocmds e.g. from plugins like vim-qf.
	let l:errorformat_saved = &l:errorformat
	call a:qf.set_errorformat()
	noautocmd execute 'caddfile' a:file
	let &l:errorformat = l:errorformat_saved
	endfunction

  unsilent echo 'DBG3' a:file
  noautocmd execute 'caddfile' a:file

On my end, the output look like this:

DBG2 /home/lervag/.local/plugged/vimtex/test/issues/2144/dir\ with\ spaces/test.pplatex

Notice how the spaces are escaped with a backslash. The backslashes are added by fnameescape, which I believe should also work on Windows and add the necessary escapes. But I may be wrong.

[lervag]

Can you test if things work if you change to the following:

  let l:tmp = fnamemodify(a:log, ':r') . '.pplatex'
  let l:log = a:log

  silent call system(printf('pplatex -i "%s" >"%s"', l:log, l:tmp))

(Remove the fnameescapes and then add "..."s in the pplatex command line.)

[BertrandSim]

Edit: I did not place the unsilent echos in the right way yesterday, here is the updated output.

I also tested on filepaths which have no spaces (as a control).

Could you add some unsilent echos as specified below and report what they show when you have spaces in the paths?

1. For the file with no spaces (control):

DBG1 pplatex -i C:\Users\Bertrand\Desktop\a1/orient_metric.log >C:\Users\Bertrand\Desktop\a1/orient_metric.pplatex
DBG2 C:\Users\Bertrand\Desktop\a1/orient_metric.pplatex
DBG3 C:\Users\Bertrand\Desktop\a1/orient_metric.pplatex
DBG3 C:\Users\Bertrand\Desktop\a1/orient_metric.blg

2. For a file with spaces (eg, changing "orient_metric" to "orient metric",

DBG1 pplatex -i C:\Users\Bertrand\Desktop\a1/orient\ metric.log >C:\Users\Bertrand\Desktop\a1/orient\ metric.pplatex
DBG2 C:\Users\Bertrand\Desktop\a1/orient\ metric.pplatex

, and the qf window did not open (an error).

Can you test if things work if you change to the following:

3. For a file with spaces, with fnameescapes removed:

DBG1 pplatex -i "C:\Users\Bertrand\Desktop\a1/orient metric.log" >"C:\Users\Bertrand\Desktop\a1/orient metric.pplatex"
DBG2 C:\Users\Bertrand\Desktop\a1/orient metric.pplatex
DBG3 C:\Users\Bertrand\Desktop\a1/orient metric.pplatex
DBG3 C:\Users\Bertrand\Desktop\a1/orient\ metric.blg

This works! The qf window was shown.

---

Now, here is where things get interesting.

4. When "orient metric.pplatex" is present in the directory, the output for case 2 is quite different (file with spaces, fnameescapes present):

DBG1 pplatex -i C:\Users\Bertrand\Desktop\a1/orient\ metric.log >C:\Users\Bertrand\Desktop\a1/orient\ metric.pplatex
DBG2 C:\Users\Bertrand\Desktop\a1/orient\ metric.pplatex
DBG3 C:\Users\Bertrand\Desktop\a1/orient\ metric.pplatex
DBG3 C:\Users\Bertrand\Desktop\a1/orient\ metric.blg

This works. The qf window was shown.
In addition, the .pplatex file remained in the directory (despite https://github.com/lervag/vimtex/blob/master/autoload/vimtex/qf/pplatex.vim#L92).

(After deleting the .pplatex file manually, there was no debugging output, as before.)

[BertrandSim]

Also, changing the 'shellslash' setting made no difference to the above output

[lervag]

For a file with spaces, with fnameescapes removed:
...
This works! The qf window was shown.

It's interesting that in your examples, the path is C:\Users\Bertrand\Desktop\a1/..., i.e. it suddenly uses a forward slash. It does not seem to matter, as it seems my proposed change actually does resolve the issue?

[BertrandSim]

I found the mix of / and \ strange as well (not sure why that is the case). In any case, yes, wrapping filepaths in "..." in place of fnameescape does resolve it. Thank you :)