Commits

Branch selector

 python-3-upgrade

User selector

All users

All time

Commit History

Commits on Sep 29, 2021

• Update README

  sebastian-nagel

  committed
  09fdeb2

  Copy full SHA for 09fdeb2

• Upgrade to Python 3: build_local_zipnum.py

  sebastian-nagel

  committed
  ce45503

  Copy full SHA for ce45503

Commits on Sep 28, 2021

• Make logging work for dosample.py

  Show description for 347228b
  sebastian-nagel

  committed
  347228b

  Copy full SHA for 347228b

Commits on Sep 27, 2021

• Upgrade to Python 3:

  Show description for 2e0e210
  sebastian-nagel

  committed
  2e0e210

  Copy full SHA for 2e0e210

• Upgrade to Python 3:

  Show description for d72d621
  sebastian-nagel

  committed
  d72d621

  Copy full SHA for d72d621

Commits on Dec 17, 2019

• Upgrade to mrjob 0.7.0

  Show description for af341bb
  sebastian-nagel

  committed
  af341bb

  Copy full SHA for af341bb

Commits on Nov 25, 2019

• Set public-read ACL during cdx-*.gz upload

  sebastian-nagel

  committed
  23a8daf

  Copy full SHA for 23a8daf

Commits on Oct 29, 2019

• Replace s3cmd by AWS CLI

  sebastian-nagel

  committed
  1fe10d4

  Copy full SHA for 1fe10d4

Commits on Apr 16, 2019

• Add missing import

  sebastian-nagel

  committed
  7a3d1bf

  Copy full SHA for 7a3d1bf

Commits on Feb 28, 2019

• SampleCDXJob:

  Show description for eef0f5d
  sebastian-nagel

  committed
  eef0f5d

  Copy full SHA for eef0f5d

Commits on Jan 16, 2019

• Improve script to create index on Hadoop

  sebastian-nagel

  committed
  a0431b3

  Copy full SHA for a0431b3

Commits on May 2, 2018

• - obligatorily adjust S3 access control / permissions

  Show description for 5a99573
  sebastian-nagel

  committed
  5a99573

  Copy full SHA for 5a99573

Commits on Feb 23, 2018

• Upgrade for use with PyWB 2.0

  sebastian-nagel

  committed
  bcd78be

  Copy full SHA for bcd78be

• Upgrade to use boto3:

  Show description for 56a03b8
  sebastian-nagel

  committed
  56a03b8

  Copy full SHA for 56a03b8

Commits on Jan 29, 2018

• Upgrade to use boto3:

  Show description for fa2fc8b
  sebastian-nagel

  committed
  fa2fc8b

  Copy full SHA for fa2fc8b

• Replace s3cmd by AWS CLI, set public-read ACL when creating objects

  sebastian-nagel

  committed
  4b797ad

  Copy full SHA for 4b797ad

Commits on Jan 9, 2018

• Upgrade mrjob option processing (from optparse to argparse), fixes #4

  sebastian-nagel

  committed
  7a76abb

  Copy full SHA for 7a76abb

Commits on Sep 28, 2017

• Increase task memory (use default) for sample jobs

  sebastian-nagel

  committed
  ce88b55

  Copy full SHA for ce88b55

Commits on Jul 3, 2017

• Improved script to publish index

  Show description for 377d72c
  sebastian-nagel

  committed
  377d72c

  Copy full SHA for 377d72c

Commits on Mar 3, 2017

• fix syntax error

  sebastian-nagel

  committed
  1cf422d

  Copy full SHA for 1cf422d

Commits on Dec 15, 2016

• Explicitely do not compress output of sample job, fixes #2,

  Show description for 0afe5bf
  sebastian-nagel

  committed
  0afe5bf

  Copy full SHA for 0afe5bf

• proper error message if WARC file is not found

  sebastian-nagel

  committed
  9977d92

  Copy full SHA for 9977d92

Commits on Dec 1, 2016

• Add script to publish the index on s3://commoncrawl/

  sebastian-nagel

  committed
  cdf4603

  Copy full SHA for cdf4603

• Fix property to compress map output

  sebastian-nagel

  committed
  870cca7

  Copy full SHA for 870cca7

Commits on Nov 3, 2016

• Shell script to run all steps on Hadoop:

  Show description for b841ae8
  sebastian-nagel

  committed
  b841ae8

  Copy full SHA for b841ae8

• Save the job output also as binary text

  Show description for 966cd88
  sebastian-nagel

  committed
  966cd88

  Copy full SHA for 966cd88

Commits on Oct 30, 2016

• use boto to upload cdx to S3, improve logging

  sebastian-nagel

  committed
  514670c

  Copy full SHA for 514670c

• Shell script to run all steps on Hadoop

  sebastian-nagel

  committed
  b74f4f5

  Copy full SHA for b74f4f5

Commits on Oct 5, 2016

• Set mapreduce.job.reduce.slowstart.completedmaps to 1.0: with few mappers being slow when reading from S3,

  Show description for dfe3fa0
  sebastian-nagel

  committed
  dfe3fa0

  Copy full SHA for dfe3fa0

Commits on Sep 15, 2016

• add note how to install requirements in Hadoop cluster

  sebastian-nagel

  committed
  70c71ed

  Copy full SHA for 70c71ed

Commits on Sep 14, 2016

• adapt scripts to be run on Hadoop clusters without EMR

  sebastian-nagel

  committed
  ededee0

  Copy full SHA for ededee0

• add bootstrap script for EC2 instances on EMR

  sebastian-nagel

  committed
  904ce9f

  Copy full SHA for 904ce9f

Commits on May 17, 2016

• use os.remove() instead of shutil.remove() to delete local file

  sebastian-nagel

  committed
  cf19ab2

  Copy full SHA for cf19ab2

Commits on Apr 8, 2016

• Updated S3 paths after migration (from aws-publicdatasets to commoncrawl)

  mrt

  committed
  2199677

  Copy full SHA for 2199677

Commits on Feb 22, 2016

• indexwarcsjob.py: update hard-coded path to be correct (TODO: load from env)

  Show description for 0ee5018
  ikreymer

  committed
  0ee5018

  Copy full SHA for 0ee5018