WARC writer: consider usage of Type 7 UUIDs as WARC-Record-ID

[sebastian-nagel]

Common Crawl's WarcWriter uses a type 4 (pseudo randomly generated) UUID generated by Java's UUID.randomUUID() method.

RFC 9562 (published in 2024 and updating RFC 4122) defines a new Type 7 combining a Unix timestamp (epoch seconds) with random. This would allow to encode the capture time (WARC-Date) in the UUID used in the WARC-Record-Id

• adding usable information to the record ID (e.g., for verification of the WARC-Date)
  • this requires that the capture time is used as timestamp and not the time the WARC record is created
• while reducing the entropy required to store the WARC-Record-ID, at least, in the columnar index (see Add column to hold WARC-Record-ID cc-index-table#42)
• sorts by timestamp
• uniqueness should be still guaranteed by the 74 random bits (or 62 bit random plus a 12 bit sub-millisecond timestamp)

Notes and links:

• https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_7_(timestamp_and_random)
• https://belief-driven-design.com/uuid-v7-java-3ccbb/
  • code: https://github.com/belief-driven-design/blog-uuidv7 (no license)
• https://fastuuid.com/learn-about-uuids/uuid-version-7
• the new UUID type 7 has recently been implemented in OpenJDK for Java 26, see
  • JDK-8334015
  • UUID.html#ofEpochMillis(long) (early access Javadocs)

  • a Unix timestamp in milliseconds in the most significant 48 bits, allocating the required version (4 bits) and variant (2-bits) and filling the remaining 74 bits with random bits.

• https://mvnrepository.com/artifact/com.fasterxml.uuid/java-uuid-generator
  • code: https://github.com/cowtowncoder/java-uuid-generator (Apache licensed)
  • Javadoc: https://javadoc.io/static/com.fasterxml.uuid/java-uuid-generator/5.1.0/com/fasterxml/uuid/Generators.html
• other WARC writers seem to use mostly (or exclusively) the type 4 random UUID:
  • jwarc
  • webarchive-commons
  • warcio (since 1.5.2)
  • https://docs.rs/warc/latest/warc/struct.Record.html#method.generate_record_id

[wumpus]

If uniqueness is guaranteed with a shorter UUID, why didn't we always use a shorter UUID? 😜 (edit: this is mostly a joke)

Being ordered might be helpful for speeding up lookups in parquet files. Looks like it can accelerate sqlite, too.

[sebastian-nagel]

why didn't we always use a shorter UUID? 😜

We did not have a time machine to fly into 2024 and bring RFC 9562 back together with type 7. The other UUID types are based on time and MAC address which did not guarantee uniqueness in a usual crawler setup. See webrecorder/warcio@575d16f.

[lfoppiano]

Note: the timestamp should be the fetch time