WarcCdxWriter: extraction of redirect targets for CDX should not be case-sensitive

[sebastian-nagel]

If the HTTP header "Location" is written not in this form (eg. all letters lowercase) it is not extracted and added to the CDX as field "redirect". Of course, the extraction should be case-insensitive.

Seen in CC-MAIN-2020-24, to be fixed in CC-MAIN-2020-33 (August crawl). Partially fixed (not for all segments) in CC-MAIN-2020-29.

Example:

• CDX:

{
  "status": "301",
  "url": "http://1001moviespodcast.libsyn.com/episode-25-the-great-escape-1963",
  "filename": "crawl-data/CC-MAIN-2020-24/segments/1590347417746.33/crawldiagnostics/CC-MAIN-20200601113849-20200601143849-00536.warc.gz",
  "offset": "18536",
  "length": "3691"
}

• HTTP header from WARC:

HTTP/1.1 301 Moved Permanently
server: Apache/2.2.15 (CentOS)
location: https://1001moviespodcast.libsyn.com/episode-25-the-great-escape-1963
...

[sebastian-nagel]

Also affects HTTP header "Content-Type" when writing CDX and as support for charset detector (part of language detector).

[sebastian-nagel]

Fixed with b3b78bb.