See https://bugzilla.mozilla.org/show_bug.cgi?id=1738781 for context.
At first I thought that the test-case there behaving suboptimally was a Firefox bug, but after more investigation I think it's a Chromium bug instead (see https://bugzilla.mozilla.org/show_bug.cgi?id=1738781#c2 and following).
I think we could consider invalidating the anchor after applying an adjustment if when we selected the anchor there were dirty children. This will guarantee a better anchor regardless of focus moving or what not. Blink implements something like that already for content-visibility: auto.
cc @chrishtr @vmpstr