Commits

Branch selector

 master

User selector

All users

All time

Commit History

Commits on May 5, 2026

• Upgrade to ia-web-commons 3.0.5-SNAPSHOT

  Show description for f530475
  sebastian-nagel

  committed
  f530475

  Copy full SHA for f530475

Commits on Dec 3, 2025

• Upgrade to ia-web-commons 3.0.3-SNAPSHOT

  Show description for 5a39046
  sebastian-nagel

  committed
  5a39046

  Copy full SHA for 5a39046

Commits on Aug 28, 2025

• Merge pull request #11 from commoncrawl/upgrade-webarchive-commons-3.0.0

  Show description for 3a740ce
  sebastian-nagel

  authored
  3a740ce

  Copy full SHA for 3a740ce

Commits on Aug 26, 2025

• Dependency upgrades: slf4j and commons-collections

  sebastian-nagel

  committed
  c1c6cb0

  Copy full SHA for c1c6cb0

• Upgrade to ia-web-commons 3.0.1-SNAPSHOT

  Show description for b533536
  sebastian-nagel

  committed
  b533536

  Copy full SHA for b533536

Commits on Jan 8, 2025

• Upgrade dependencies

  Show description for 60332cd
  sebastian-nagel

  committed
  60332cd

  Copy full SHA for 60332cd

Commits on Nov 30, 2024

• Upgrade dependencies

  Show description for 1de1df1
  sebastian-nagel

  committed
  1de1df1

  Copy full SHA for 1de1df1

Commits on Nov 26, 2024

• Upgrade dependencies

  Show description for d2673a6
  sebastian-nagel

  committed
  d2673a6

  Copy full SHA for d2673a6

Commits on Oct 18, 2024

• Merge pull request #5 from commoncrawl/4-skip-unneeded-functionality-dependencies

  Show description for f5027e9
  sebastian-nagel

  authored
  f5027e9

  Copy full SHA for f5027e9

Commits on Oct 7, 2024

• Remove custom Maven repositories

  sebastian-nagel

  committed
  fb0b053

  Copy full SHA for fb0b053

• Remove unneeded classes and their dependencies

  Show description for da349bd
  sebastian-nagel

  committed
  da349bd

  Copy full SHA for da349bd

Commits on Mar 18, 2024

• fix broken link in pom file

  

  thunderpoot

  authored and

  sebastian-nagel

  committed
  e5c89e8

  Copy full SHA for e5c89e8

Commits on Oct 6, 2023

• Merge pull request #6 from commoncrawl/weat-generator-catch-exceptions

  Show description for 00fa0c7
  jnioche

  authored
  00fa0c7

  Copy full SHA for 00fa0c7

• WEATGenerator: catch exceptions thrown by ExtractingResourceProducer,

  Show description for 599eac6
  sebastian-nagel

  committed
  599eac6

  Copy full SHA for 599eac6

Commits on Jul 20, 2020

• WAT/WET generator: log warning but do not NPE if input path does not exist

  sebastian-nagel

  committed
  a97e384

  Copy full SHA for a97e384

Commits on Jun 5, 2020

• WET extractor: add identified natural language of text content

  Show description for 2b7f735
  sebastian-nagel

  committed
  2b7f735

  Copy full SHA for 2b7f735

Commits on Nov 21, 2019

• Reduce logging when transforming WARC to WAT/WET, fixes #3

  sebastian-nagel

  committed
  75200d0

  Copy full SHA for 75200d0

Commits on Oct 22, 2019

• Upgrade dependencies

  Show description for 47baba5
  sebastian-nagel

  committed
  47baba5

  Copy full SHA for 47baba5

• Use openjson to replace org.json

  Show description for 8da3563
  sebastian-nagel

  committed
  8da3563

  Copy full SHA for 8da3563

Commits on May 4, 2017

• Upgrade dependency to ia-web-commons to 1.1.9-SNAPSHOT

  Show description for 49c811f
  sebastian-nagel

  committed
  49c811f

  Copy full SHA for 49c811f

Commits on Apr 28, 2017

• Upgrade dependency to ia-web-commons to 1.1.8-SNAPSHOT

  Show description for cd18a3a
  sebastian-nagel

  committed
  cd18a3a

  Copy full SHA for cd18a3a

Commits on Sep 14, 2016

• Fix compilation error on OpenJDK 7

  sebastian-nagel

  committed
  325bb9a

  Copy full SHA for 325bb9a

Commits on Jun 26, 2014

• add missing return to skipExisting

  Aloisius

  committed
  a67afd7

  Copy full SHA for a67afd7

• Allow args to be order insensitive. Should just use real arg parsing.

  Aloisius

  committed
  3d470f7

  Copy full SHA for 3d470f7

• Add skipExisting flag so that we can skip over warc files which already have wat & wet files outputted for them.

  Aloisius

  committed
  3627a11

  Copy full SHA for 3627a11

• Revert "Skip over records we can't parse"

  Show description for 28f31bb
  Aloisius

  committed
  28f31bb

  Copy full SHA for 28f31bb

• Skip over records we can't parse

  Aloisius

  committed
  2d37906

  Copy full SHA for 2d37906

Commits on Jun 23, 2014

• add collections dependency

  Aloisius

  committed
  26ef94b

  Copy full SHA for 26ef94b

Commits on Mar 19, 2014

• Log every 1000 rows, and update usage to be accurate.

  Aloisius

  committed
  bf597ae

  Copy full SHA for bf597ae

Commits on Mar 18, 2014

• Use counters instead of log messages to make it a bit easier to track

  Aloisius

  committed
  58ecde3

  Copy full SHA for 58ecde3

Commits on Jan 2, 2014

• Always write out wet/wat files relative to the warc file

  Aloisius

  committed
  46ed268

  Copy full SHA for 46ed268

Commits on Nov 26, 2013

• Print out something useful if -strictMode is used

  Aloisius

  committed
  ca34131

  Copy full SHA for ca34131

• Shut up the gzip debugging and reorder wet/wat generation since wat deletes our text node

  Aloisius

  committed
  ae8198c

  Copy full SHA for ae8198c

• Add ourselves to the job driver

  Aloisius

  committed
  bbd5b82

  Copy full SHA for bbd5b82

• First pass at a WAT + WET generator that uses's CCs forked ia-web-commons

  Aloisius

  committed
  cbd7766

  Copy full SHA for cbd7766