Create modern jobs api for simple processes

[lervag]

NB! This PR is not finished yet and is not ready for testing.

I want to remove the old process.vim module and switch it with a new jobs.vim module that relies on the jobstart (nvim) and job_start (Vim) functions. I believe this may be useful in many cases, and perhaps give a better experience on Windows especially (cf. e.g. #1885).

[lervag]

This is now more or less ready for testing. @clason, I would be happy if you could test this on MacOS. I believe it should work more or less exactly as before (you should see no difference), so testing would only be to look for inconsistencies or errors.

I would also be glad if anyone could test this on Windows.

[lervag]

I'll be so bold as to ask some of the recent issue contributors that seem to use Windows: @habamax @Bongbang

[habamax]

Sure, will check it, @lervag

[habamax]

regarding #1885, I am experiencing the same delay for initial <C-x><C-o> (~30 ~40 seconds for my doc). Not sure if you were to fix it with this PR too.

Other than that it works the same for me. Will continue using it and let you know if there are issues.

[lervag]

regarding #1885, I am experiencing the same delay for initial <C-x><C-o> (~30 ~40 seconds for my doc). Not sure if you were to fix it with this PR too.

Well, actually, I was hoping that this might resolve the delay, at least in part. :\

Other than that it works the same for me. Will continue using it and let you know if there are issues.

That's good to hear, though! Are you on Vim or neovim?

[habamax]

Another note, after I VimtexClearCache All, it takes < 2 secs to show the completion.

Tried with another new document it is < 2 seconds. :)

Not sure why first time was so long.

[habamax]

That's good to hear, though! Are you on Vim or neovim?

Latest nightly vim (gVim)

[lervag]

it takes < 2 secs to show the completion.

That sounds much better than 30 - 40 seconds. Or do I misunderstand anything?

[habamax]

That sounds much better than 30 - 40 seconds. Or do I misunderstand anything?

Now with cleared cache it is < 2 seconds, all consequent calls are instant.

[lervag]

How is this compared to the current master branch? Is the master branch still slow (as in O(10) seconds)?

[habamax]

How is this compared to the current master branch? Is the master branch still slow (as in O(10) seconds)?

Current master is about 10 seconds for cleared cache.

[lervag]

Cool; so, if I understand correctly, this PR should improve the experience for Windows users.

I'm pushing another update now, then I'll wait for some more feedback before I merge. This does change an essential part, so I believe it is sensible to avoid rushing the merge...

[clason]

This is now more or less ready for testing. @clason, I would be happy if you could test this on MacOS. I believe it should work more or less exactly as before (you should see no difference), so testing would only be to look for inconsistencies or errors.

I did a brief test, and everything seems to work as intended; I haven't done a proper side-by-side comparison, but it's certainly not slower and quite probably faster! (I'm on an M1 at the moment, so everything feels very snappy already ;))

[lervag]

Great, thanks for testing! I don't think it should be much faster, except perhaps on Windows where I've removed some silly hacks. But I hope it will be more robust, and there may be some performance improvements due to more consistent code.

[lervag]

I'm tempted to merge this. Feel free to dissuade me, I'll wait until later today in any case.

[lervag]

Thanks; I've merged this, then. :)

[lervag]

I've also pushed quite a few updates that further improve on this. I've tested quite thoroughly on my side with neovim and Vim, and I believe things should still work well on MacOS and Windows. However, I am curious to hear feedback on this, especially from the Windows side (@habamax).

[lervag]

Btw, some very crude benchmarking from my side:

• Neovim (hyperfine -M 3 make)

  commit	Timing (s)	Note
  fb7c6d5	48.187 ± 0.221	Latest version
  f40305e	49.068 ± 0.079
  ef3b014	48.965 ± 0.031	This PR merged
  ef3b014	49.217 ± 0.213
  aaecdbb	48.282 ± 0.189	Before this PR

• Vim (MYVIM="vim -T dumb --not-a-term -n" hyperfine -M 3 make)

  commit	Timing (s)	Note
  fb7c6d5	51.026 ± 0.138	Latest version
  f40305e	52.302 ± 0.138
  ef3b014	52.239 ± 0.204	This PR merged
  ef3b014	52.208 ± 0.153
  aaecdbb	51.368 ± 0.097	Before this PR

The commands are run inside VIMTEX/test; hyperfine is here. Again, this is very crude and does not really say very much. Between aaecdbb and fb7c6d5 there are also added several new tests, for instance, so in a way one should expect the run time to increase. I believe these tests may (perhaps) be better than nothing for justifying a claim that the improvements are improvements, especially considering that I think they strongly improve the robustness of the code.

[tomvam]

Sorry for being late to the party. I got the email rather late to begin with and was not quite savvy enough with git and Github to download a branch for testing. (I normally just let Vundle handle the updates.)

For future reference (so that I can do my part), is the following roughly what I should do?

1. cd into vimfiles/bundle
2. git clone https://github.com/lervag/vimtex/ --branch <branch> --single-branch vimtex-test
3. Find a handy way to switch back and forth between two versions of VimTeX (maybe you can tell me).

Thank you, Karl Lervåg , for ever diligent improvements. So far, I haven't noticed much difference in the latest version except perhaps one bug that I've been meaning to report may have already be solved.

[habamax]

@lervag well, now it takes more time with cleared cache (~ 30 secs) to show the completion for the same document :)

But when I save it under other path/name -- it is again "fast enough"

I wonder if it is possible to run background job once tex file is loaded to "construct" what is behind that cache.

[lervag]

@Bongbang

Sorry for being late to the party.

No problem!

For future reference (so that I can do my part), is the following roughly what I should do?

Yes, more or less. I would do it like this:

cd /path/to/vimtex
# e.g. ~/.vim/bundle/vimtex

# Pull latest updates, including all branches
git pull --all

# Checkout the test branch
git checkout <branch>

# Checkout back to master
git checkout master

This way, you don't need to work with two copies of the code. git checkout as such serves as a handy way to switch between two versions.

Thank you, Karl Lervåg , for ever diligent improvements. So far, I haven't noticed much difference in the latest version except perhaps one bug that I've been meaning to report may have already be solved.

Great, I'm glad to hear this might have solved a bug, and I was not really expecting to hear of any noticable differences. Thank you for testing! And for the kind words :)

[lervag]

@habamax

well, now it takes more time with cleared cache (~ 30 secs) to show the completion for the same document :)

Yes, I was actually sort of expecting this. For some reason, it seems the system(...) call is much quicker than the job_start(...) call. However, I also believe the "improvement" in the previous version was faulty, because I added a manual wait for the job_start with a timeout of 5 seconds. When timing out, the system call would essentially return nothing. Thus, this did not fix anything, it only "treated the symptom".

But when I save it under other path/name -- it is again "fast enough"

What do you mean here? "save under other path/name"?

I wonder if it is possible to run background job once tex file is loaded to "construct" what is behind that cache.

Perhaps. Could I propose that you again open a new issue for this? I'm sorry about the inconvenience, but I now know much more about what actually happens, and so we could continue the discussion and possibly find a way to fix it.

One thing I'm thinking: Are you perhaps using WSL 2? Could it have something to do with the slowness of accessing files outside the linux partition?

[habamax]

But when I save it under other path/name -- it is again "fast enough"

What do you mean here? "save under other path/name"?

:w ~/docs/anotherpath/anothername.tex then switch to it and do all the same

I wonder if it is possible to run background job once tex file is loaded to "construct" what is behind that cache.

Perhaps. Could I propose that you again open a new issue for this? I'm sorry about the inconvenience, but I now know much more about what actually happens, and so we could continue the discussion and possibly find a way to fix it.

Sure will create it.

One thing I'm thinking: Are you perhaps using WSL 2? Could it have something to do with the slowness of accessing files outside the linux partition?

Well, that is another issue I might create. For this particular case I have used gVim (and in general for tex I use it).

I also have WSL1 where completion works "instantly", but the issue is that compilation is not working there, vimtex can't find latemxk executable:

[lervag]

Sure will create it.

Great, let's continue this discussion there, then.