warc-truncation/cc-main-2018-43-single-warc-file.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Truncated Records in WARC Files (October 2018)\n",
    "\n",
    "Content payload in Common Crawl archives is truncated if the content exceeds a limit of\n",
    "- [500 kiB in 2008 – 2012 ARC files](https://groups.google.com/d/topic/common-crawl/hQTRnWahcHA/discussion)\n",
    "- 1 MiB in WARC files (since 2013)\n",
    "\n",
    "The truncation is required to keep the crawl archives at a limited size and ensure that a broad sample of web pages is covered. It also avoids that the archives are filled by accidentally captured video or audio streams. The crawler needs to buffer the content temporarily and a limit ensures that this is possible with a limited amount of RAM for many parallel connections.\n",
    "\n",
    "\n",
    "While the mission of Common Crawl has always been to provide a broad sample of HTML pages, there have been always users interested in the various other document formats: PDF, office documents, etc.  Recently, a discussion started in the [Common Crawl forum](https://groups.google.com/forum/#!forum/common-crawl) to [increase the 1 MiB content limit](https://groups.google.com/d/topic/common-crawl/JJW6fv1rUQw/discussion). It also turned out that there are issues regarding the marking of truncated payloads using the [WARC-Truncated header](https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#warc-truncated), see [cc/nutch#10](//github.com/commoncrawl/nutch/issues/10).\n",
    "\n",
    "\n",
    "Metrics shown here are based on a single WARC file ([CC-MAIN-20181016221847-20181017003347-00114.warc.gz](https://data.commoncrawl.org/crawl-data/CC-MAIN-2018-43/segments/1539583510893.26/warc/CC-MAIN-20181016221847-20181017003347-00114.warc.gz)) of the October 2018 crawl. Truncated records are extracted using cc-pyspark, see [cc-pyspark#9](//github.com/commoncrawl/cc-pyspark/pull/9).\n",
    "\n",
    "The length limit is applied to the binary content after it the HTTP protocol-level content and transfer encoding have been removed (decoded). Every WARC record consists of WARC headers, HTTP headers and content payload and is as a whole compressed using gzip. This leads to a couple of length metrics (bytes):\n",
    "- `warc_record_length`: WARC compressed record length (WARC headers, HTTP headers and content payload)\n",
    "- `warc_content_length`: uncompressed length of HTTP headers and content payload (obligatorily given by `Content-Lenght` WARC header)\n",
    "- `http_content_length` resp. `payload_length`: the actual size of the payload\n",
    "- `http_orig_content_length`: the original value of the HTTP `Content-Length` header (if present) which indicates the entire length with protocol-level compression applied\n",
    "\n",
    "But let's now inspect the set of truncated records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas\n",
    "\n",
    "data = pandas.read_csv('data/truncated-records-CC-MAIN-20181015080248-20181015101748-00033.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since there was a bug which caused that not all truncated records are marked, the table also includes records with a payload length equal to the content limit (1 MiB). These are presumably truncated as the probability that a page/document has exactly a length of 1 MiB shouldn't be high. It appears that even the majority of truncated records isn't marked appropriately:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NaN           262\n",
       "length        105\n",
       "disconnect     46\n",
       "Name: truncated_reason, dtype: int64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['truncated_reason'].value_counts(dropna=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's only look at pages/documents truncated because of the length limit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tr = data[data['payload_length'] == 2**20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and see which content types are truncated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "text/html                                  210\n",
       "application/pdf                             88\n",
       "application/xhtml+xml                       37\n",
       "image/jpeg                                   9\n",
       "application/xml                              6\n",
       "audio/mpeg                                   4\n",
       "application/x-dosexec                        2\n",
       "application/vnd.android.package-archive      1\n",
       "image/png                                    1\n",
       "application/octet-stream                     1\n",
       "audio/mp4                                    1\n",
       "application/x-rar-compressed                 1\n",
       "video/webm                                   1\n",
       "application/x-msdownload; format=pe32        1\n",
       "image/bmp                                    1\n",
       "application/gzip                             1\n",
       "application/zip                              1\n",
       "video/x-msvideo                              1\n",
       "Name: identified_payload_type, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tr['identified_payload_type'].value_counts(dropna=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because Common Crawl uses gzipped WARC files as primary archive format, it's actually more the size of the gzip-compressed WARC records which matters, not the size of the uncompressed content payload. Let's put the compressed and uncompressed lengths into relation for the various content types:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>warc_record_length</th>\n",
       "      <th>warc_content_length</th>\n",
       "      <th>payload_length</th>\n",
       "      <th>count</th>\n",
       "      <th>% ratio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>identified_payload_type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>application/gzip</th>\n",
       "      <td>1.000004</td>\n",
       "      <td>1.000328</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>99.967586</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>application/octet-stream</th>\n",
       "      <td>1.001059</td>\n",
       "      <td>1.000244</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>100.081424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>application/pdf</th>\n",
       "      <td>80.557208</td>\n",
       "      <td>88.033373</td>\n",
       "      <td>88.0</td>\n",
       "      <td>88</td>\n",
       "      <td>91.507579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>application/vnd.android.package-archive</th>\n",
       "      <td>0.978990</td>\n",
       "      <td>1.000364</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>97.863308</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>application/x-dosexec</th>\n",
       "      <td>1.955378</td>\n",
       "      <td>2.000984</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>97.720791</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>application/x-msdownload; format=pe32</th>\n",
       "      <td>0.940312</td>\n",
       "      <td>1.000352</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>93.998160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>application/x-rar-compressed</th>\n",
       "      <td>0.994845</td>\n",
       "      <td>1.000288</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>99.455895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>application/xhtml+xml</th>\n",
       "      <td>3.651897</td>\n",
       "      <td>37.018148</td>\n",
       "      <td>37.0</td>\n",
       "      <td>37</td>\n",
       "      <td>9.865154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>application/xml</th>\n",
       "      <td>0.261678</td>\n",
       "      <td>6.001943</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6</td>\n",
       "      <td>4.359884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>application/zip</th>\n",
       "      <td>1.001128</td>\n",
       "      <td>1.000469</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>100.065868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>audio/mp4</th>\n",
       "      <td>0.999113</td>\n",
       "      <td>1.000328</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>99.878542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>audio/mpeg</th>\n",
       "      <td>3.852148</td>\n",
       "      <td>4.001248</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4</td>\n",
       "      <td>96.273655</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>image/bmp</th>\n",
       "      <td>0.007612</td>\n",
       "      <td>1.000273</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.761015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>image/jpeg</th>\n",
       "      <td>8.859035</td>\n",
       "      <td>9.003714</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9</td>\n",
       "      <td>98.393128</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>image/png</th>\n",
       "      <td>0.998478</td>\n",
       "      <td>1.000340</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>99.813811</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>text/html</th>\n",
       "      <td>36.280820</td>\n",
       "      <td>210.454672</td>\n",
       "      <td>210.0</td>\n",
       "      <td>210</td>\n",
       "      <td>17.239256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>video/webm</th>\n",
       "      <td>0.998388</td>\n",
       "      <td>1.000245</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>99.814365</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>video/x-msvideo</th>\n",
       "      <td>0.963777</td>\n",
       "      <td>1.000331</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>96.345776</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         warc_record_length  \\\n",
       "identified_payload_type                                       \n",
       "application/gzip                                   1.000004   \n",
       "application/octet-stream                           1.001059   \n",
       "application/pdf                                   80.557208   \n",
       "application/vnd.android.package-archive            0.978990   \n",
       "application/x-dosexec                              1.955378   \n",
       "application/x-msdownload; format=pe32              0.940312   \n",
       "application/x-rar-compressed                       0.994845   \n",
       "application/xhtml+xml                              3.651897   \n",
       "application/xml                                    0.261678   \n",
       "application/zip                                    1.001128   \n",
       "audio/mp4                                          0.999113   \n",
       "audio/mpeg                                         3.852148   \n",
       "image/bmp                                          0.007612   \n",
       "image/jpeg                                         8.859035   \n",
       "image/png                                          0.998478   \n",
       "text/html                                         36.280820   \n",
       "video/webm                                         0.998388   \n",
       "video/x-msvideo                                    0.963777   \n",
       "\n",
       "                                         warc_content_length  payload_length  \\\n",
       "identified_payload_type                                                        \n",
       "application/gzip                                    1.000328             1.0   \n",
       "application/octet-stream                            1.000244             1.0   \n",
       "application/pdf                                    88.033373            88.0   \n",
       "application/vnd.android.package-archive             1.000364             1.0   \n",
       "application/x-dosexec                               2.000984             2.0   \n",
       "application/x-msdownload; format=pe32               1.000352             1.0   \n",
       "application/x-rar-compressed                        1.000288             1.0   \n",
       "application/xhtml+xml                              37.018148            37.0   \n",
       "application/xml                                     6.001943             6.0   \n",
       "application/zip                                     1.000469             1.0   \n",
       "audio/mp4                                           1.000328             1.0   \n",
       "audio/mpeg                                          4.001248             4.0   \n",
       "image/bmp                                           1.000273             1.0   \n",
       "image/jpeg                                          9.003714             9.0   \n",
       "image/png                                           1.000340             1.0   \n",
       "text/html                                         210.454672           210.0   \n",
       "video/webm                                          1.000245             1.0   \n",
       "video/x-msvideo                                     1.000331             1.0   \n",
       "\n",
       "                                         count     % ratio  \n",
       "identified_payload_type                                     \n",
       "application/gzip                             1   99.967586  \n",
       "application/octet-stream                     1  100.081424  \n",
       "application/pdf                             88   91.507579  \n",
       "application/vnd.android.package-archive      1   97.863308  \n",
       "application/x-dosexec                        2   97.720791  \n",
       "application/x-msdownload; format=pe32        1   93.998160  \n",
       "application/x-rar-compressed                 1   99.455895  \n",
       "application/xhtml+xml                       37    9.865154  \n",
       "application/xml                              6    4.359884  \n",
       "application/zip                              1  100.065868  \n",
       "audio/mp4                                    1   99.878542  \n",
       "audio/mpeg                                   4   96.273655  \n",
       "image/bmp                                    1    0.761015  \n",
       "image/jpeg                                   9   98.393128  \n",
       "image/png                                    1   99.813811  \n",
       "text/html                                  210   17.239256  \n",
       "video/webm                                   1   99.814365  \n",
       "video/x-msvideo                              1   96.345776  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tr = tr[['warc_record_length', 'warc_content_length', 'payload_length', 'identified_payload_type']]\n",
    "tr['count'] = 1\n",
    "r = tr.groupby(['identified_payload_type']).sum()\n",
    "r['% ratio'] = 100.0*r['warc_record_length']/r['warc_content_length']\n",
    "for col in ['warc_record_length', 'warc_content_length', 'payload_length']:\n",
    "    r[col] = r[col]/2**20 # show aggregated length in MiB\n",
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `warc_record_length` also includes the WARC headers while `warc_content_length` does not. That's why the compression ratio could be a little bit too large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}