VimTex info error

[rafaelferracini]

Description

I've just installed VimTex in my Neovim environment (configured with Lua and Packer), and to check if the installation went well, I executed :VimtexInfo.
I believe it might be due to my computer's language settings (Portuguese-BR), but I can't change it due to Windows limitations.
Here's the error:

Error detected while processing function vimtex#info#open[2]..vimtex#scratch#new[2]..218[32]..220[4]..206[1]..207[1]..<SNR>88_get_os_info[18]..vimtex#jobs#cached[4]..49[4]..50:
line   6:
E474: String "<c6>o do sistema operacional:             10.0.22621 N/A compila<87><c6>o 22621" contains byte that does not start any UTF-8 character
Erro detectado ao processar function vimtex#info#open[2]..vimtex#scratch#new[2]..218[32]..220[4]..206:
line    1:
E1098: String, List or Blob required

[lervag]

I believe it might be due to my computer's language settings (Portuguese-BR), but I can't change it due to Windows limitations.

I've tried to fix this. Can you update to the latest version and try again and let me know how it goes?

I don't expect the output to be perfect here, but I hope that it looks good enough and that there is no error.

[rafaelferracini]

I believe it might be due to my computer's language settings (Portuguese-BR), but I can't change it due to Windows limitations.

I've tried to fix this. Can you update to the latest version and try again and let me know how it goes?

I don't expect the output to be perfect here, but I hope that it looks good enough and that there is no error.

Same error but now a different string

Erro detectado ao processar function vimtex#info#open[2]..vimtex#scratch#new[2]..218[32]..220[4]..206[1]..207[1]..<SNR>88_get_os_info[18]..vimtex#jobs#cached[4]..5[4]..6:

linha    6:
E474: String "<a0>rio registrado:                  [my email adress here]" contains byte that does not start any UTF-8 character
Erro detectado ao processar function vimtex#info#open[2]..vimtex#scratch#new[2]..218[32]..220[4]..206:
linha    1:
E1098: String, List or Blob required

<a0>rio probably refers to "Usuário", translations for "User"

[user202729]

Line 4 of vimtex#jobs#cached is

function! vimtex#jobs#cached(cmd) abort " {{{1
  " Cached version of vimtex#jobs#capture(...)
  let l:cache = vimtex#cache#open('capture')

>  return l:cache.has(a:cmd)
        \ ? l:cache.get(a:cmd)
        \ : l:cache.set(a:cmd, vimtex#jobs#capture(a:cmd))
endfunction

annoyingly the following functions's names are not displayed by vim, so I can't tell which function the following line 4 and line 6 is of. Neither can I tell you're using persistent or temporary cache. (persistent is the default.)

Try typing

:echo vimtex#cache#path("capture.json")

and read the content of the JSON file at the output, see what's the content.

---

Edit: I didn't realize earlier, but the error is caused by neovim trying to store the string into the JSON file. https://github.com/neovim/neovim/blob/1cf51a07a6bfb827efc36911cd018da3a3cb863b/src/nvim/eval/encode.c#L633

[lervag]

annoyingly the following functions's names are not displayed by vim

I'm also annoyed by this fact, although I've become quite adapt at recognizing functions from script numbers and similar.

[lervag]

Same error but now a different string

Ok, it seems I was onto something, but I'm not quite sure. And it's hard, because I can't really reproduce this properly. Not sure how to properly catch all of this without somehow writing a full windows-1252 to utf-8 converter...

[lervag]

Oh, and I would also need to properly detect the encoding. These things are hard.

[user202729]

Some quick testing:

let a={}
function A_abc()
	echoerr 123
endfunction
let a.abc=function('A_abc')
call a.abc()

seems to work well enough at displaying the function name. Could anything go wrong?

Edit: perhaps dict property is needed, :h Dictionary-function.

[user202729]

That code looks... strange (and without much rationale what's going on). Is it just that if match(x, '[\x80-\x9F,\xE1]') > 0 then it's considered "likely" that the thing is invalid UTF8 (and thus valid Windows-1252)?

Then the next question is, why would systeminfo command output in Windows-1252 encoding instead of UTF-8...? Is there a way to simply make it output in UTF-8 (which would avoid the conversion)?

---

Edit: I didn't realize earlier, but neovim throws this error while trying to store such a string in JSON

https://github.com/neovim/neovim/blob/1cf51a07a6bfb827efc36911cd018da3a3cb863b/src/nvim/eval/encode.c#L633

Another thing, the code doesn't look like valid Windows-1252 either.

In [1]: b"\x87\xc6".decode('u8')
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
Cell In[1], line 1
----> 1 b"\x87\xc6".decode('u8')

File /usr/lib/python3.11/encodings/utf_8.py:16, in decode(input, errors)
     15 def decode(input, errors='strict'):
---> 16     return codecs.utf_8_decode(input, errors, True)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 0: invalid start byte

In [2]: b"\x87\xc6".decode('windows-1252')
Out[2]: '‡Æ'

and neither is it valid UTF-8.

The word compila<87><c6>o probably should be compilação. But I can't find any encoding that makes \x87 maps to ç and \xc6 maps to ã.

[lervag]

Some quick testing: …

Sorry, but I don't see quite how this is relevant. The problem here seems related to the encoding of strings passed as arguments to functions. In particular, Vimscript (perhaps also valid for Lua code in neovim) relies on UTF-8 encoded strings and will raise an error if it is somehow passed a string with bytecodes that are not valid UTF-8 bytecodes.

I may have understood things wrong, though.

That code looks... strange (and without much rationale what's going on).

I should probably write a comment here to explain!

Is it just that if match(x, '[\x80-\x9F,\xE1]') > 0 then it's considered "likely" that the thing is invalid UTF8 (and thus valid Windows-1252)?

Yes, exactly.

Then the next question is, why would systeminfo command output in Windows-1252 encoding instead of UTF-8...? Is there a way to simply make it output in UTF-8 (which would avoid the conversion)?

Because systeminfo will output text encoded in whatever is specified in the system regional settings. E.g., I believe the standard encoding for Portuguese-BR is Windows 1252. @hrcHarry from #2850 reported the exact same error, so this affects multiple people.

I'm slightly inclined to just ignore it, but at the same time I feel it should be possible to find a relatively simple solution here.

Edit: I didn't realize earlier, but neovim throws this error while trying to store such a string in JSON

https://github.com/neovim/neovim/blob/1cf51a07a6bfb827efc36911cd018da3a3cb863b/src/nvim/eval/encode.c#L633

Oh, nice! Thanks for going deeper! It seems obvious now that I should have checked the source code for this error message!

Another thing, the code doesn't look like valid Windows-1252 either.

I agree to this. When debugging, I also found it did not seem to look right when decoded as windows-1252. So it is possible I'm wrong about the source encoding here. But I think the end result is that I need to somehow remove or convert bytes in some byterange to avoid this error.

The word compila<87><c6>o probably should be compilação. But I can't find any encoding that makes \x87 maps to ç and \xc6 maps to ã.

Perhaps the problematic bytecode is a result of parsing only part of some sequence, thus that the original code has some starting part swallowed before the error appears?

[lervag]

But OK, the problem is clearly related to json_encode used here:

vimtex/autoload/vimtex/cache.vim

Line 218 in e7ce03e

call writefile([json_encode(self.data)], self.path)