WARC writer (CDX writer): new optional CDX JSON fields "redirect" and "truncated"

[sebastian-nagel]

• add key "truncated" if the record payload is truncated indication the reason for the truncation, cf.
  WARC-Truncated in WARC 1.1 spec
• add key "redirect" containing the redirect target
  • from HTTP header field "Location" if the HTTP status code indicates a HTTP redirect
  • relative paths converted to absolute URLs using the page URL as base/context
  • absent if the "Location" value is missing or is not a valid URL or a valid relative URL path

Example CDX snippets (multi-line JSON):

• redirect target/location

org,commoncrawl)/faq 20191107134157 {
  "url": "https://commoncrawl.org/faq/",
  ...,
  "status": "301",
  ...,
  "redirect": "http://commoncrawl.org/big-picture/frequently-asked-questions/"
}

• truncation because of overlong content

es,remax,inmomas)/robots.txt 20191107134158 {
  "url": "http://www.inmomas.remax.es/robots.txt",
  ...,
  "status": "200",
  ...,
  "truncated": "length"
}