Commits

Branch selector

 main

User selector

All users

All time

Commit History

Commits on Jan 27, 2026

• Merge pull request #8 from commoncrawl/spark-zipnumcluster

  Show description for 2c7b102
  sebastian-nagel

  authored
  2c7b102

  Copy full SHA for 2c7b102

• refactor: increase driver memory and max. result size

  Show description for 8a404af
  sebastian-nagel

  committed
  8a404af

  Copy full SHA for 8a404af

Commits on Jan 10, 2026

• refactor: adapt publish_index.sh for Spark jobs

  sebastian-nagel

  committed
  d995a58

  Copy full SHA for d995a58

Commits on Nov 19, 2025

• Spark jobs to build index: README, add shell script and requirements

  sebastian-nagel

  committed
  af10f62

  Copy full SHA for af10f62

• fix: suppress ignored CLI options

  sebastian-nagel

  committed
  a9ff183

  Copy full SHA for a9ff183

• fix: unify logging, address Pylint warnings

  sebastian-nagel

  committed
  beac6c8

  Copy full SHA for beac6c8

• refactor: persistence and temporary files

  Show description for 02be757
  sebastian-nagel

  committed
  02be757

  Copy full SHA for 02be757

• fix: use correct timestamp in cluster.idx

  sebastian-nagel

  committed
  8394c74

  Copy full SHA for 8394c74

• refactor: code documentation, formatting and cleanup

  sebastian-nagel

  committed
  7b0f5cc

  Copy full SHA for 7b0f5cc

• fix: type error when using partition boundary file

  Show description for 136f1ae
  sebastian-nagel

  committed
  136f1ae

  Copy full SHA for 136f1ae

• fix: do not create output directory tree in local filesystem

  sebastian-nagel

  committed
  64a2e11

  Copy full SHA for 64a2e11

• refactor: move functions out of global scope

  Show description for 7779a47
  sebastian-nagel

  committed
  7779a47

  Copy full SHA for 7779a47

Commits on Nov 14, 2025

• refactor: rename Python files to valid module names

  sebastian-nagel

  committed
  0ce27c8

  Copy full SHA for 0ce27c8

• fix: sort by SURT key *and* capture timestamp

  sebastian-nagel

  committed
  014d702

  Copy full SHA for 014d702

Commits on Nov 13, 2025

• fix: code formatting and comments

  sebastian-nagel

  committed
  a1f8553

  Copy full SHA for a1f8553

Commits on Jan 27, 2025

• chore: adding TODO for upcoming tasks

  jt55401

  committed
  3f60a21

  Copy full SHA for 3f60a21

Commits on Dec 20, 2024

• fix: remove keyfunc, it isn't needed (and was causing issue that sort was only working on first char

  jt55401

  committed
  fcd02cf

  Copy full SHA for fcd02cf

• fix: marking args as required

  jt55401

  committed
  91a18e7

  Copy full SHA for 91a18e7

• fix: spelling of boundaries

  jt55401

  committed
  30b3ce7

  Copy full SHA for 30b3ce7

Commits on Dec 16, 2024

• fix: bugfixes found from integration testing

  jt55401

  committed
  0f0b6d4

  Copy full SHA for 0f0b6d4

Commits on Dec 3, 2024

• chore: going back to using CCFileProcessorSparkJob

  jt55401

  committed
  c0c454b

  Copy full SHA for c0c454b

Commits on Dec 2, 2024

• fix: temporary working version of zipnum job using normal spark features (need to refactor ccpyspark back into it now)

  jt55401

  committed
  55f907c

  Copy full SHA for 55f907c

Commits on Nov 27, 2024

• fix: more emr bugfixes and tweaks

  jt55401

  committed
  1d9a857

  Copy full SHA for 1d9a857

• fix: numerous bugfixes from complex merge issues

  jt55401

  committed
  9c0f47b

  Copy full SHA for 9c0f47b

• chore: merging other changes

  jt55401

  committed
  1ce6690

  Copy full SHA for 1ce6690

• chore: merged latest normal spark solution upwards

  jt55401

  committed
  7299693

  Copy full SHA for 7299693

• fix: get zipnumcluster-pyspark working with s3/emr

  jt55401

  committed
  688e879

  Copy full SHA for 688e879

Commits on Nov 20, 2024

• fix: final cluster.idx logic

  jt55401

  committed
  3f876a8

  Copy full SHA for 3f876a8

• fix: fix cdx filenames

  jt55401

  committed
  d3f5ee9

  Copy full SHA for d3f5ee9

• fix: getting order and structure of final index proper, and in single file

  jt55401

  committed
  951ede4

  Copy full SHA for 951ede4

• fix: bugfix for edge conditions

  jt55401

  committed
  2a1de31

  Copy full SHA for 2a1de31

• fix: re-init and flush gz

  jt55401

  committed
  e8dcf78

  Copy full SHA for e8dcf78

• fix: simpler ordered partition id

  jt55401

  committed
  4a99403

  Copy full SHA for 4a99403

• feat: reservoir sampled method of zipnum cluster job

  jt55401

  committed
  e6bb313

  Copy full SHA for e6bb313

Commits on Nov 8, 2024

• docs: adding note about zipnum testing and behaviour

  jt55401

  committed
  fcb5695

  Copy full SHA for fcb5695