Commits

Branch selector

 master

User selector

All users

All time

Commit History

Commits on Mar 12, 2026

• Merge pull request #7 from commoncrawl/upgrade-java-11

  Show description for 1d780a1
  sebastian-nagel

  authored
  1d780a1

  Copy full SHA for 1d780a1

• Merge pull request #8 from commoncrawl/sample-link-alternate-hreflang

  Show description for 24c1a5c
  parismic

  authored
  24c1a5c

  Copy full SHA for 24c1a5c

• WATSampleOutlinks: sample translations from <link rel=alternate hreflang=...>

  Show description for a070214
  sebastian-nagel

  committed
  a070214

  Copy full SHA for a070214

• Integrate GitHub CI build

  sebastian-nagel

  committed
  a2e69fa

  Copy full SHA for a2e69fa

• Apply spotless code formatting rules

  sebastian-nagel

  committed
  a0b8db0

  Copy full SHA for a0b8db0

• Integrate spotless code formatter

  sebastian-nagel

  committed
  2f0e5af

  Copy full SHA for 2f0e5af

• Fix broken logo link

  sebastian-nagel

  committed
  e25c2b2

  Copy full SHA for e25c2b2

• Require minimum Java version 11

  Show description for eecb7d6
  sebastian-nagel

  committed
  eecb7d6

  Copy full SHA for eecb7d6

Commits on Dec 17, 2024

• Merge pull request #5 from commoncrawl/wat-multivalue-headers

  Show description for 5aea741
  sebastian-nagel

  authored
  5aea741

  Copy full SHA for 5aea741

Commits on Dec 13, 2024

• Adapt WATSampleOutlinks job to read multi-valued headers in WAT data

  Show description for 20eca18
  sebastian-nagel

  committed
  20eca18

  Copy full SHA for 20eca18

• Adapt WATServerType example to read multi-valued headers in WAT data

  Show description for 66bc8bf
  sebastian-nagel

  committed
  66bc8bf

  Copy full SHA for 66bc8bf

• Improve TagCounterMap

  Show description for 4b269de
  sebastian-nagel

  committed
  4b269de

  Copy full SHA for 4b269de

• Modernize examples

  Show description for 30dd7a7
  sebastian-nagel

  committed
  30dd7a7

  Copy full SHA for 30dd7a7

• Upgrade dependencies

  Show description for 4204cbe
  sebastian-nagel

  committed
  4204cbe

  Copy full SHA for 4204cbe

Commits on Jul 9, 2024

• fix(WATSampleOutLinks): use Text.getBytes() following API docs

  Show description for fabf5ca
  sebastian-nagel

  committed
  fabf5ca

  Copy full SHA for fabf5ca

Commits on May 24, 2023

• Allow to specify a bot name when search for "nofollow" HTML metadata

  Show description for 0eedb4a
  sebastian-nagel

  committed
  0eedb4a

  Copy full SHA for 0eedb4a

• Allow to specify a bot name when search for "nofollow" HTML metadata

  Show description for e7bfb1c
  sebastian-nagel

  committed
  e7bfb1c

  Copy full SHA for e7bfb1c

Commits on Jan 6, 2023

• Upgrade dependencies

  Show description for c64d3c8
  sebastian-nagel

  committed
  c64d3c8

  Copy full SHA for c64d3c8

• Use "nofollow" rel attributes and HTML/HTTP metadata when

  Show description for 33e0bbd
  sebastian-nagel

  committed
  33e0bbd

  Copy full SHA for 33e0bbd

Commits on Apr 6, 2022

• Implement new methods to access data on s3://commoncrawl/

  Show description for dbd6670
  sebastian-nagel

  committed
  dbd6670

  Copy full SHA for dbd6670

Commits on Sep 24, 2021

• WATSampleOutlinks:

  Show description for 04a8c43
  sebastian-nagel

  committed
  04a8c43

  Copy full SHA for 04a8c43

Commits on May 21, 2021

• Upgrade dependencies

  Show description for bf87278
  sebastian-nagel

  committed
  bf87278

  Copy full SHA for bf87278

• WATSampleOutlinks:

  Show description for a7992c5
  sebastian-nagel

  committed
  a7992c5

  Copy full SHA for a7992c5

Commits on May 20, 2021

• Remove obsolete jar (webarchive-commons dependency managed by Maven)

  sebastian-nagel

  committed
  ba959cd

  Copy full SHA for ba959cd

Commits on May 3, 2021

• WATSampleOutlinks:

  Show description for 17bfa39
  sebastian-nagel

  committed
  17bfa39

  Copy full SHA for 17bfa39

Commits on Jan 21, 2020

• WATSampleOutlinks:

  Show description for 8badc44
  sebastian-nagel

  committed
  8badc44

  Copy full SHA for 8badc44

Commits on Apr 9, 2019

• WATSampleOutlinks:

  Show description for 8e6a55a
  sebastian-nagel

  committed
  8e6a55a

  Copy full SHA for 8e6a55a

Commits on Mar 28, 2018

• WATSampleOutlinks:

  Show description for 7f87e6a
  sebastian-nagel

  committed
  7f87e6a

  Copy full SHA for 7f87e6a

Commits on Nov 14, 2017

• WATSampleOutlinks:

  Show description for 14aa2d8
  sebastian-nagel

  committed
  14aa2d8

  Copy full SHA for 14aa2d8

Commits on Oct 14, 2017

• Add tool to extract and sample outlinks from WAT files

  sebastian-nagel

  committed
  cb355fa

  Copy full SHA for cb355fa

Commits on Oct 9, 2017

• Update libraries (Hadoop and webarchive-commons)

  sebastian-nagel

  committed
  f1747e9

  Copy full SHA for f1747e9

Commits on Jul 28, 2017

• Update Common Crawl logo location

  sebastian-nagel

  committed
  e220036

  Copy full SHA for e220036

Commits on Feb 23, 2017

• Merge branch 'tfmorris-hadoop2'

  sebastian-nagel

  committed
  14f298d

  Copy full SHA for 14f298d

• Update Github links & Java version

  

  tfmorris

  authored and

  sebastian-nagel

  committed
  55743c8

  Copy full SHA for 55743c8

• Update to Hadoop 2

  Show description for b21b550
  

  tfmorris

  authored and

  sebastian-nagel

  committed
  b21b550

  Copy full SHA for b21b550