Commits

Breadcrumbs

History for

onmaster

User selector

All users

All time

Commit History

Commits on Aug 28, 2019

• WAT: only unescape complete XML/HTML character entities (fixes #19)

  Show description for bb754a3
  sebastian-nagel

  committed
  bb754a3

  Copy full SHA for bb754a3

Commits on Aug 27, 2019

• WAT: only unescape complete XML/HTML character entities (fixes #19)

  Show description for 716eae9
  sebastian-nagel

  committed
  716eae9

  Copy full SHA for 716eae9

Commits on Aug 19, 2019

• WAT: only unescape complete XML/HTML character entities (fixes #19)

  Show description for 60b6f7e
  sebastian-nagel

  committed
  60b6f7e

  Copy full SHA for 60b6f7e

Commits on Jul 23, 2019

• Use openjson as replacement for the org.json dependency

  Show description for d188b9e
  sebastian-nagel

  committed
  d188b9e

  Copy full SHA for d188b9e

Commits on Jul 22, 2019

• Replace the org.json dependency by Ted Dunning's open-json library

  Show description for 470c353
  sebastian-nagel

  committed
  470c353

  Copy full SHA for 470c353

Commits on Jul 8, 2019

• Add unit test for HTML entity decoding

  Show description for fb514ab
  sebastian-nagel

  committed
  fb514ab

  Copy full SHA for fb514ab

• WAT/WET extraction: use compiled regex Pattern to get better performance

  sebastian-nagel

  committed
  4a1f323

  Copy full SHA for 4a1f323

• WAT: unescape XML/HTML character entities (#14)

  Show description for d5af4e5
  sebastian-nagel

  committed
  d5af4e5

  Copy full SHA for d5af4e5

Commits on Aug 23, 2017

• WAT extractor: get links from onClick attributes, fixes #8

  Show description for ea0729b
  sebastian-nagel

  committed
  ea0729b

  Copy full SHA for ea0729b

• WAT extracor: add rel attribute to A@/href links, fixes #10

  Show description for dda2c89
  sebastian-nagel

  committed
  dda2c89

  Copy full SHA for dda2c89

• WET extractor: improve spacing of textual content, fixes #13

  Show description for 7ce4e88
  sebastian-nagel

  committed
  7ce4e88

  Copy full SHA for 7ce4e88

Commits on Jun 5, 2017

• Limit pattern matching URLs embedded in CSS to match max. 8000 characters,

  Show description for ea00e6d
  sebastian-nagel

  committed
  ea00e6d

  Copy full SHA for ea00e6d

Commits on May 4, 2017

• Fix HTTP-Response-Metadata for wget WARCs. Changes came from https://github.com/commoncrawl/ia-web-commons/commit/58e85a60d75707da55ed499f836e57d49347484a

  

  ldko

  authored and

  sebastian-nagel

  committed
  3fac189

  Copy full SHA for 3fac189

Commits on Apr 28, 2017

• Do not add value of preceding HTTP header field if there is no value (or

  Show description for aad4e05
  sebastian-nagel

  committed
  aad4e05

  Copy full SHA for aad4e05

• Merge with recent master of iipc/webarchive-commons, transfer all

  Show description for 58e85a6
  sebastian-nagel

  committed
  58e85a6

  Copy full SHA for 58e85a6

Commits on Mar 21, 2017

• Merge branch 'wat-improved-link-extraction' of https://github.com/sebastian-nagel/webarchive-commons into sebastian-nagel-wat-improved-link-extraction

  ldko

  committed
  f12e86f

  Copy full SHA for f12e86f

Commits on Mar 2, 2017

• Fix #25: move missing unit tests over from Heritrix3

  MohammedElsayyed

  committed
  6aa43f8

  Copy full SHA for 6aa43f8

Commits on Feb 22, 2017

• Improve HTML link extraction

  Show description for 11579c2
  sebastian-nagel

  committed
  11579c2

  Copy full SHA for 11579c2

Commits on Feb 15, 2017

• Merge pull request #69 from sebastian-nagel/cc-url-parser-ignore-empty-port

  Show description for cb91063
  ldko

  authored
  cb91063

  Copy full SHA for cb91063

• Merge pull request #68 from sebastian-nagel/HTMLResourceFactory_to_use_CharsetDetector

  Show description for ed33bae
  ldko

  authored
  ed33bae

  Copy full SHA for ed33bae

• Merge pull request #65 from dportabella/fix

  Show description for 1ee2529
  ldko

  authored
  1ee2529

  Copy full SHA for 1ee2529

• Merge pull request #62 from iipc/kris-sigur-patch-1

  Show description for c5876b3
  ldko

  authored
  c5876b3

  Copy full SHA for c5876b3

Commits on Jan 27, 2017

• CharsetDetector: remove unnecessary check for empty string (contributed by @ldko)

  sebastian-nagel

  committed
  0384028

  Copy full SHA for 0384028

Commits on Jan 18, 2017

• CSS link extraction: clip also unpaired leading and trailing quotation marks

  sebastian-nagel

  committed
  194a1fa

  Copy full SHA for 194a1fa

Commits on Jan 4, 2017

• Improve clipping of quotation marks in CSS link extraction

  Show description for b918f7f
  sebastian-nagel

  committed
  b918f7f

  Copy full SHA for b918f7f

Commits on Dec 9, 2016

• Strip empty port, do not fail

  sebastian-nagel

  committed
  9e41abc

  Copy full SHA for 9e41abc

Commits on Nov 24, 2016

• Match http-equiv meta elements with unquoted attribute values, e.g.

  Show description for 824dd82
  sebastian-nagel

  committed
  824dd82

  Copy full SHA for 824dd82

• HTML encoding detection: fix errors with empty content or empty charset values

  sebastian-nagel

  committed
  607acaa

  Copy full SHA for 607acaa

• Use CharsetDetector to guess encoding of HTML document

  sebastian-nagel

  committed
  5f223d6

  Copy full SHA for 5f223d6

Commits on Sep 21, 2016

• fix: last header was lost if LF LF (intead of CRLF CRLF)

  dportabella

  committed
  9d7abed

  Copy full SHA for 9d7abed

Commits on Aug 7, 2016

• Make regular expression to extract URLs from CSS more restrictive

  Show description for a23cfeb
  sebastian-nagel

  committed
  a23cfeb

  Copy full SHA for a23cfeb

Commits on Jul 13, 2016

• Remove invalid constant

  Show description for 0cbca57
  

  Kristinn Sigurðsson

  authored
  0cbca57

  Copy full SHA for 0cbca57

Commits on May 26, 2016

• Merge pull request #54 from vonrosen/modify-strip-session-params

  Show description for 1485fdd
  

  Kristinn Sigurðsson

  committed
  1485fdd

  Copy full SHA for 1485fdd

• Merge pull request #50 from nlevitt/her-2089

  Show description for cf5f600
  

  Kristinn Sigurðsson

  committed
  cf5f600

  Copy full SHA for cf5f600

• Merge pull request #51 from nlevitt/fix-warc-sizeOnDisk

  Show description for f5e6541
  

  Kristinn Sigurðsson

  committed
  f5e6541

  Copy full SHA for f5e6541