Function tokenizer::preprocess 3x faster

[ayosec]

I added a bench to test the difference:

fn bench_preprocess(b: &mut test::Bencher) {
    let source = "Lorem\n\t\u{FFFD}ipusm\ndoror\u{FFFD}á\n";
    b.iter(|| {
        let _ = super::preprocess(source);
    });
}

With the old function, with 3 replace() (commit ea6c1b7):

test tokenizer::bench_preprocess::bench_preprocess  ... bench:       440 ns/iter (+/- 24)

With the new function:

test tokenizer::bench_preprocess::bench_preprocess  ... bench:       142 ns/iter (+/- 6)

I'm a newbie in Rust, so I guess that there are a lot of improvements to be done.

[SimonSapin]

I think this whole block is unnecessary. UTF-8 bytes for a multi-byte code point all in 0x80..0xFF, so copying each byte separately is fine.

[ayosec]

I added it because the byte \x0C is replaced by \n in a previous pattern, and I'm not sure if there is any multibyte character that contains \x0c.

For instance, a character like FA 0C 10 can be replaced with FA 0A 10.

I'm not a Unicode expert, though.

[SimonSapin]

UTF-8 is well designed this way: bytes in 0x00 to 0x7F can only happen in single-byte code points.

[ayosec]

Done! Thanks for the review. The code looks much better now.

[SimonSapin]

Looks great, thanks for the patch!