Commits

Breadcrumbs

History for

oncc-github-workflow

User selector

All users

All time

Commit History

Commits on Mar 10, 2024

• Update crawl documentation

  Show description for 83acd50
  

  derhecht

  authored and

  sebastian-nagel

  committed
  83acd50

  Copy full SHA for 83acd50

Commits on Dec 17, 2020

• Add possibility to setup deduplication group mode in crawl script (#557)

  derhecht

  authored
  88a17f2

  Copy full SHA for 88a17f2

Commits on May 15, 2020

• NUTCH-2496 Speed up link inversion step in crawling script

  Show description for 7fba6df
  sebastian-nagel

  committed
  7fba6df

  Copy full SHA for 7fba6df

Commits on Apr 28, 2020

• NUTCH-2495: Use -deleteGone instead of clean job in crawl script while indexing

  sebastian-nagel

  committed
  7ebd35d

  Copy full SHA for 7ebd35d

• NUTCH-2501 allow to set Java heap size when using crawl script in distributed mode

  Show description for a455eb5
  sebastian-nagel

  committed
  a455eb5

  Copy full SHA for a455eb5

• NUTCH-2501 allow to set Java heap size when using crawl script in distributed mode

  Show description for b5e794e
  sebastian-nagel

  committed
  b5e794e

  Copy full SHA for b5e794e

• NUTCH-2781 Increase default Java heap size

  Show description for 3214840
  sebastian-nagel

  committed
  3214840

  Copy full SHA for 3214840

Commits on Jan 19, 2020

• NUTCH-2759 bin/crawl: Rename option --num-slaves

  Show description for 040d71d
  sebastian-nagel

  committed
  040d71d

  Copy full SHA for 040d71d

Commits on May 6, 2019

• NUTCH-2626 bin/crawl: remove option -noParsing from fetch command

  sebastian-nagel

  committed
  290e3cb

  Copy full SHA for 290e3cb

Commits on Jul 2, 2018

• NUTCH-2510 bin/crawl: optionally update HostDb and use in generate

  Show description for 5c5f415
  

  okedoki

  authored and

  sebastian-nagel

  committed
  5c5f415

  Copy full SHA for 5c5f415

Commits on Mar 12, 2018

• fixed hdfs file checks in crawl script

  

  okedoki

  authored and

  sebastian-nagel

  committed
  0e28af6

  Copy full SHA for 0e28af6

Commits on Jan 9, 2018

• NUTCH-2493: Add new parameter to crawler script to configure sitemap processing frequency

  mfeltscher

  committed
  7085b26

  Copy full SHA for 7085b26

Commits on Jan 3, 2018

• Add notes

  mfeltscher

  committed
  b4d445f

  Copy full SHA for b4d445f

• Adjust usage output

  mfeltscher

  committed
  cce97f0

  Copy full SHA for cce97f0

• NUTCH-2492: Add more configuration parameters to crawl script

  mfeltscher

  committed
  af12547

  Copy full SHA for af12547

• Update usage information in header as well

  mfeltscher

  committed
  036ad2c

  Copy full SHA for 036ad2c

• NUTCH-2491: Integrate sitemap processing and HostDB into crawl script

  mfeltscher

  committed
  32eac63

  Copy full SHA for 32eac63

Commits on Dec 13, 2016

• fix for NUTCH-2046 contributed by jnioche

  jnioche

  committed
  7b0103f

  Copy full SHA for 7b0103f

Commits on Oct 30, 2015

• Fix for NUTCH-2150 Add ProtocolStatus Utility contributed by Michael Joyce <mltjoyce@gmail.com> this closes #82.

  Show description for 7906c26
  chrismattmann

  committed
  7906c26

  Copy full SHA for 7906c26

Commits on Oct 22, 2015

• NUTCH-2148 Review and update mapred --> mapreduce config params in crawl script

  Show description for da6f381
  lewismc

  committed
  da6f381

  Copy full SHA for da6f381

Commits on Aug 24, 2015

• NUTCH-2049 Upgrade Trunk to Hadoop > 2.4 stable

  Show description for 845b922
  jnioche

  committed
  845b922

  Copy full SHA for 845b922

Commits on Jun 25, 2015

• Adding some continuous crawl goodies to the crawl script NUTCH-2036

  Show description for b14a6d5
  jnioche

  committed
  b14a6d5

  Copy full SHA for b14a6d5

Commits on Apr 21, 2015

• Fix for NUTCH-1987 - Make bin/crawl indexer agnostic contributed by Michael Joyce <mltjoyce@gmail.com> this closes #18.

  Show description for f034163
  chrismattmann

  committed
  f034163

  Copy full SHA for f034163

Commits on Nov 11, 2014

• NUTCH-1883 in case of generate: break loop and do not exit with error

  Show description for 3b18ed0
  sebastian-nagel

  committed
  3b18ed0

  Copy full SHA for 3b18ed0

Commits on Oct 27, 2014

• NUTCH-1883 bin/crawl: use function to run bin/nutch and check exit value

  Show description for 09849b8
  sebastian-nagel

  committed
  09849b8

  Copy full SHA for 09849b8

Commits on Sep 27, 2014

• NUTCH-1853 Add commented out WebGraph executions to ./bin/crawl

  Show description for 9f9b852
  lewismc

  committed
  9f9b852

  Copy full SHA for 9f9b852

Commits on Sep 3, 2014

• Fix for NUTCH-1832: Nutch should run without an indexer.

  Show description for 6bb5684
  chrismattmann

  committed
  6bb5684

  Copy full SHA for 6bb5684

Commits on Aug 29, 2014

• NUTCH-1828 bin/crawl : incorrect handling of nutch errors

  Show description for be979bb
  jnioche

  committed
  be979bb

  Copy full SHA for be979bb

Commits on Jul 5, 2014

• NUTCH-1566 bin/nutch to allow whitespace in paths

  Show description for 282af5e
  sebastian-nagel

  committed
  282af5e

  Copy full SHA for 282af5e

Commits on Apr 17, 2014

• NUTCH-1761 Crawl script fails to find job file if not started from inside bin dir (David Hosking, jnioche)

  Show description for 2fd3821
  jnioche

  committed
  2fd3821

  Copy full SHA for 2fd3821

Commits on Nov 14, 2013

• NUTCH-656 Generic Deduplicator (jnioche, snagel)

  Show description for 3d1af7b
  jnioche

  committed
  3d1af7b

  Copy full SHA for 3d1af7b

Commits on Oct 3, 2013

• NUTCH-1639 bin/crawl fails on mac os (various contributors via snagel)

  Show description for ad6f951
  jnioche

  committed
  ad6f951

  Copy full SHA for ad6f951

Commits on Apr 18, 2013

• Fixed detection of job file for distributed mode

  Show description for 5903a6b
  jnioche

  committed
  5903a6b

  Copy full SHA for 5903a6b

Commits on Jan 15, 2013

• NUTCH-1500 bin/crawl fails on step solrindex with wrong path to segment

  Show description for 578a418
  sebastian-nagel

  committed
  578a418

  Copy full SHA for 578a418

Commits on Jul 10, 2012

• NUTCH-1087 Deprecate crawl command and replace with example script

  Show description for 39da6c1
  jnioche

  committed
  39da6c1

  Copy full SHA for 39da6c1

Loading