Commits

Branch selector

 enable-integrity

User selector

All users

All time

Commit History

Commits on Aug 4, 2024

• fix up paths/naming per Sebastian's comments

  jt55401

  committed
  e734d5b

  Copy full SHA for e734d5b

• Fixing a few java bugs

  jt55401

  committed
  10f91c0

  Copy full SHA for 10f91c0

Commits on Jul 22, 2024

• Added CDX generation (-outputCDX) to the WEATGenerator - may not compile yet, local env not available today.

  jt55401

  committed
  42f6479

  Copy full SHA for 42f6479

Commits on Jul 2, 2024

• laying down TODO's for integrity integration

  jt55401

  committed
  cb6188f

  Copy full SHA for cb6188f

Commits on Mar 18, 2024

• fix broken link in pom file

  

  thunderpoot

  authored and

  sebastian-nagel

  committed
  e5c89e8

  Copy full SHA for e5c89e8

Commits on Oct 6, 2023

• Merge pull request #6 from commoncrawl/weat-generator-catch-exceptions

  Show description for 00fa0c7
  jnioche

  authored
  00fa0c7

  Copy full SHA for 00fa0c7

• WEATGenerator: catch exceptions thrown by ExtractingResourceProducer,

  Show description for 599eac6
  sebastian-nagel

  committed
  599eac6

  Copy full SHA for 599eac6

Commits on Jul 20, 2020

• WAT/WET generator: log warning but do not NPE if input path does not exist

  sebastian-nagel

  committed
  a97e384

  Copy full SHA for a97e384

Commits on Jun 5, 2020

• WET extractor: add identified natural language of text content

  Show description for 2b7f735
  sebastian-nagel

  committed
  2b7f735

  Copy full SHA for 2b7f735

Commits on Nov 21, 2019

• Reduce logging when transforming WARC to WAT/WET, fixes #3

  sebastian-nagel

  committed
  75200d0

  Copy full SHA for 75200d0

Commits on Oct 22, 2019

• Upgrade dependencies

  Show description for 47baba5
  sebastian-nagel

  committed
  47baba5

  Copy full SHA for 47baba5

• Use openjson to replace org.json

  Show description for 8da3563
  sebastian-nagel

  committed
  8da3563

  Copy full SHA for 8da3563

Commits on May 4, 2017

• Upgrade dependency to ia-web-commons to 1.1.9-SNAPSHOT

  Show description for 49c811f
  sebastian-nagel

  committed
  49c811f

  Copy full SHA for 49c811f

Commits on Apr 28, 2017

• Upgrade dependency to ia-web-commons to 1.1.8-SNAPSHOT

  Show description for cd18a3a
  sebastian-nagel

  committed
  cd18a3a

  Copy full SHA for cd18a3a

Commits on Sep 14, 2016

• Fix compilation error on OpenJDK 7

  sebastian-nagel

  committed
  325bb9a

  Copy full SHA for 325bb9a

Commits on Jun 26, 2014

• add missing return to skipExisting

  Aloisius

  committed
  a67afd7

  Copy full SHA for a67afd7

• Allow args to be order insensitive. Should just use real arg parsing.

  Aloisius

  committed
  3d470f7

  Copy full SHA for 3d470f7

• Add skipExisting flag so that we can skip over warc files which already have wat & wet files outputted for them.

  Aloisius

  committed
  3627a11

  Copy full SHA for 3627a11

• Revert "Skip over records we can't parse"

  Show description for 28f31bb
  Aloisius

  committed
  28f31bb

  Copy full SHA for 28f31bb

• Skip over records we can't parse

  Aloisius

  committed
  2d37906

  Copy full SHA for 2d37906

Commits on Jun 23, 2014

• add collections dependency

  Aloisius

  committed
  26ef94b

  Copy full SHA for 26ef94b

Commits on Mar 19, 2014

• Log every 1000 rows, and update usage to be accurate.

  Aloisius

  committed
  bf597ae

  Copy full SHA for bf597ae

Commits on Mar 18, 2014

• Use counters instead of log messages to make it a bit easier to track

  Aloisius

  committed
  58ecde3

  Copy full SHA for 58ecde3

Commits on Jan 2, 2014

• Always write out wet/wat files relative to the warc file

  Aloisius

  committed
  46ed268

  Copy full SHA for 46ed268

Commits on Nov 26, 2013

• Print out something useful if -strictMode is used

  Aloisius

  committed
  ca34131

  Copy full SHA for ca34131

• Shut up the gzip debugging and reorder wet/wat generation since wat deletes our text node

  Aloisius

  committed
  ae8198c

  Copy full SHA for ae8198c

• Add ourselves to the job driver

  Aloisius

  committed
  bbd5b82

  Copy full SHA for bbd5b82

• First pass at a WAT + WET generator that uses's CCs forked ia-web-commons

  Aloisius

  committed
  cbd7766

  Copy full SHA for cbd7766

Commits on Nov 25, 2013

• programdriver-workaround hadoop 1.x & 2.x incompat

  vinaygoel

  committed
  b5498ab

  Copy full SHA for b5498ab

Commits on Nov 23, 2013

• Extract info from warc/metadata records (outlinks/hopinfo)

  vinaygoel

  committed
  964c9a7

  Copy full SHA for 964c9a7

• Tools to generate WAT and CDX files for (W)ARCs stored in HDFS

  vinaygoel

  committed
  ef89dcb

  Copy full SHA for ef89dcb

Commits on Nov 20, 2013

• Merge branch 'master' of github.com:internetarchive/ia-hadoop-tools

  ikreymer

  committed
  881d3ba

  Copy full SHA for 881d3ba

• cdxtosstable: support for gzip, don't log dupes

  ikreymer

  committed
  f863d8a

  Copy full SHA for f863d8a

• add cassandra partitioner as pig is having trouble finding the standard one

  Show description for 97c74b9
  ikreymer

  committed
  97c74b9

  Copy full SHA for 97c74b9

• Merge branch 'master' of github.com:internetarchive/ia-hadoop-tools

  ikreymer

  committed
  c9fe3a3

  Copy full SHA for c9fe3a3