WarcRecordWriter performance improvements

[sebastian-nagel]

According to async-profiler, when writing WARC files the CPU time is mostly spent for:

• gzip compression
• SHA digests
• language detection
• charset detection

(interactive SVG.zip)

Tasks to improve the performance:

• improve performance charset detection, see Speedup initialization of charset AutoDetectReader required for language detection #7
• avoid unnecessary recoding from UTF-8 to UTF-8, verify validity instead

[tfmorris]

• Since the vast majority of pages are in UTF-8 encoding, decoding then reencoding them seems very wasteful.
• It might be useful to compare the performance of the ICU charset detector or Google's vs Tika's
• The entire payload doesn't need to be scanned for character set detection. A few KB should be plenty (I think ICU defaults to 8 KB).

[sebastian-nagel]

* decoding then reencoding

Unfortunately, CLD2 may segfault when the input is not 100% valid UTF-8. That's worse because it will kill the task and we'll loose data and stop the crawling. I haven't yet looked for a method to only validate UTF-8, reencoding was just the simplest way in terms of coding. If you know one let me know.

* [ICU charset detector](http://userguide.icu-project.org/conversion/detection)

Tika uses a patched version of the ICU detector internally. The main improvement: it uses also the declared encoding as an additional hint while ICU dectector ignores it (ICU-8737 - won't fix)

Google's
Thanks for the pointer, I wasn't aware of it. If time I test it. Would also require some work to prepare the Java native bindings and compile and deploy the lib in the cluster.

* The entire payload doesn't need to be scanned  for character set detection. A few KB should be plenty (I think ICU defaults to 8 KB).

Tika's detectors set limits internally (HtmlEncodingDetector and txt.CharsetDetector).

[sebastian-nagel]

vast majority of pages are in UTF-8 encoding, decoding then reencoding them seems very wasteful

Thanks, @tfmorris. Guava provides a method to validate UTF-8. Calling it beforehand (2e5b0b9) and re-encoding only invalid UTF-8 (invalid UTF-8 sequences are replaced when decoding to Java string) saves some time - roughly, validating UTF-8 takes only 20-25% of the time required for re-encoding.
In a single WARC file among 55,271 records there are 47,057 HTML pages encoded in UTF-8 and 617 of them fail to validate.

[sebastian-nagel]

Profile with the improvements in production: fetcher.reduce.17657.async-prof.svg.zip

I don't see any options for substantial improvements

• 18% language detection: 13% CLD2, 3% Tika charset detection
• 37% gzip compression, mostly spent in libz.so
• 16% SHA-1 digests: since the WARC spec explicitly does not recommend any hash algorithm, maybe the better solution is to switch to another algorithm in the future.

[chris-ha458]

• 37% gzip compression, mostly spent in libz.so
• 16% SHA-1 digests: since the WARC spec explicitly does not recommend any hash algorithm, maybe the better solution is to switch to another algorithm in the future.

Replacing gzip with zst could reduce compression time by atleast 50%. The savings are likely more due to zst leveraging multiprocessing.
migrating away from SHA-1 to SHA-2 would cost around 10% more though.
Overall still faster

• ( Compression 37% -> 18%, Digests 16% -> 18%. Sum 53% -> 36%)

If it is migrated to BLAKE3, that would be cut down to around 25%.
Overall bit more faster

• ( Compression 37% -> 18%, Digests 16% -> 4%. Sum 53% -> 22%)