fix: Drop invalid UTF-8 scanner candidates

[mvanhorn]

Summary

Replace the three unchecked scanner conversions in crates/oxide/src/scanner/mod.rs with checked conversions that omit only extracted slices that are not valid UTF-8. Apply the check in the shared extract pipeline so initial scans, incremental scan_content calls, and CSS-variable extraction cannot insert invalid strings into scanner state, and apply the same policy to both branches of get_candidates_with_positions while preserving byte offsets and the legacy -[] restoration. Keep the extractor's byte-oriented CSS identifier classification unchanged: accepting non-ASCII bytes during extraction is useful for valid multibyte code points, while the conversion boundary is the authoritative place to enforce the String contract.

The scanner currently converts extracted byte slices with unchecked UTF-8 constructors at the shared extraction boundary and both candidate-with-position branches. A source file containing a stray continuation byte can therefore produce an invalid String, violating Rust's string invariant and allowing corrupted candidates to persist in a long-lived scanner. The thread provides a deterministic reproduction using invalid bytes inside an arbitrary value, so the problem no longer depends on reproducing the originally reported Turbopack race. Valid candidates found alongside malformed byte sequences must continue to be returned normally.

Fixes #20368

Test plan

Not applicable to this change.

AI was used for assistance.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8d5140ef-4d5e-4e32-9581-a6e707c5b2e3

📥 Commits

Reviewing files that changed from the base of the PR and between 3524b45 and 517f8ce.

📒 Files selected for processing (2)

• crates/oxide/src/scanner/mod.rs
• crates/oxide/tests/scanner.rs

---

Walkthrough

The scanner replaced unsafe UTF-8 conversions with checked conversions. Invalid UTF-8 candidates are discarded during full, incremental, and generic extraction. Tests cover invalid candidate filtering, valid and Unicode candidate preservation, position tracking, and CSS variable extraction with invalid UTF-8 input.

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: dropping invalid UTF-8 candidates from the scanner.
Description check	✅ Passed	The description directly explains the checked UTF-8 conversions, affected scanner paths, preserved behavior, and tests.
Linked Issues check	✅ Passed	The changes satisfy issue #20368 by filtering invalid UTF-8 candidates while preserving valid candidates, offsets, and restoration behavior.
Out of Scope Changes check	✅ Passed	The code and tests are directly related to the linked issue and stated scanner UTF-8 safety objectives.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; malformed UTF-8 candidates are omitted without changing extraction, restoration, or position semantics for surviving candidates.

The checked conversions operate independently on extracted tokens, cover persistent scanner-state and positioned-candidate paths, and preserve the existing byte-offset and legacy restoration logic.

_{Reviews (1): Last reviewed commit: "fix: address self-review findings" | Re-trigger Greptile}

[thejamesgore]

Saw this was created after my fix comment #20368 on the issue so I ran the repro harness against this branch to check it closes what I found there.

On main 4be6110 the four invalid-byte cases fail as before, lone 0x80, 0xFE, 0xFF, and a truncated E2 80 all come back as ill formed strings. On this branch 517f8ce all three pass and the valid-ASCII control still returns its candidates and the full oxide scanner suite is green with 67 passed so this verifabily fixes the soundness issue including the deterministic repro.

Two things worth flagging while you're here though:

1. The fuzzer still can't catch a regression of this. fuzz_targets/parsing.rs gates on if let Ok(s) = std::str::from_utf8(data) meaning only the &str path is ever fuzzed so the raw-byte path that fs::read feeds that this PR fixes never gets exercised. Feeding the fuzz target raw bytes would keep this class of bug from coming back silently.

2. The other thing is dropped candidates now disappear with no diagnostic and since this isn't limited to the Turbopack race it means any Latin-1/Windows-1252 source file triggers it and a user with a legacy encoded file will just see utilities silently not generating so a tracing WARN event at the conveersation boundary would make that debuggable.