Commits

Breadcrumbs

History for

onmaster

User selector

All users

All time

Commit History

Commits on Jul 8, 2019

• Add unit test for HTML entity decoding

  Show description for fb514ab
  sebastian-nagel

  committed
  fb514ab

  Copy full SHA for fb514ab

• WAT/WET extraction: use compiled regex Pattern to get better performance

  sebastian-nagel

  committed
  4a1f323

  Copy full SHA for 4a1f323

• WAT: unescape XML/HTML character entities (#14)

  Show description for d5af4e5
  sebastian-nagel

  committed
  d5af4e5

  Copy full SHA for d5af4e5

Commits on Aug 23, 2017

• WAT extractor: get links from onClick attributes, fixes #8

  Show description for ea0729b
  sebastian-nagel

  committed
  ea0729b

  Copy full SHA for ea0729b

• WAT extracor: add rel attribute to A@/href links, fixes #10

  Show description for dda2c89
  sebastian-nagel

  committed
  dda2c89

  Copy full SHA for dda2c89

• WET extractor: improve spacing of textual content, fixes #13

  Show description for 7ce4e88
  sebastian-nagel

  committed
  7ce4e88

  Copy full SHA for 7ce4e88

Commits on Jun 5, 2017

• Limit pattern matching URLs embedded in CSS to match max. 8000 characters,

  Show description for ea00e6d
  sebastian-nagel

  committed
  ea00e6d

  Copy full SHA for ea00e6d

Commits on May 4, 2017

• Fix HTTP-Response-Metadata for wget WARCs. Changes came from https://github.com/commoncrawl/ia-web-commons/commit/58e85a60d75707da55ed499f836e57d49347484a

  

  ldko

  authored and

  sebastian-nagel

  committed
  3fac189

  Copy full SHA for 3fac189

Commits on Apr 28, 2017

• Do not add value of preceding HTTP header field if there is no value (or

  Show description for aad4e05
  sebastian-nagel

  committed
  aad4e05

  Copy full SHA for aad4e05

• Merge with recent master of iipc/webarchive-commons, transfer all

  Show description for 58e85a6
  sebastian-nagel

  committed
  58e85a6

  Copy full SHA for 58e85a6

Commits on Mar 21, 2017

• Merge branch 'wat-improved-link-extraction' of https://github.com/sebastian-nagel/webarchive-commons into sebastian-nagel-wat-improved-link-extraction

  ldko

  committed
  f12e86f

  Copy full SHA for f12e86f

Commits on Mar 2, 2017

• Fix #25: move missing unit tests over from Heritrix3

  MohammedElsayyed

  committed
  6aa43f8

  Copy full SHA for 6aa43f8

Commits on Feb 22, 2017

• Improve HTML link extraction

  Show description for 11579c2
  sebastian-nagel

  committed
  11579c2

  Copy full SHA for 11579c2

Commits on Feb 15, 2017

• Merge pull request #69 from sebastian-nagel/cc-url-parser-ignore-empty-port

  Show description for cb91063
  ldko

  authored
  cb91063

  Copy full SHA for cb91063

• Merge pull request #68 from sebastian-nagel/HTMLResourceFactory_to_use_CharsetDetector

  Show description for ed33bae
  ldko

  authored
  ed33bae

  Copy full SHA for ed33bae

• Merge pull request #65 from dportabella/fix

  Show description for 1ee2529
  ldko

  authored
  1ee2529

  Copy full SHA for 1ee2529

• Merge pull request #62 from iipc/kris-sigur-patch-1

  Show description for c5876b3
  ldko

  authored
  c5876b3

  Copy full SHA for c5876b3

Commits on Jan 27, 2017

• CharsetDetector: remove unnecessary check for empty string (contributed by @ldko)

  sebastian-nagel

  committed
  0384028

  Copy full SHA for 0384028

Commits on Jan 18, 2017

• CSS link extraction: clip also unpaired leading and trailing quotation marks

  sebastian-nagel

  committed
  194a1fa

  Copy full SHA for 194a1fa

Commits on Jan 4, 2017

• Improve clipping of quotation marks in CSS link extraction

  Show description for b918f7f
  sebastian-nagel

  committed
  b918f7f

  Copy full SHA for b918f7f

Commits on Dec 9, 2016

• Strip empty port, do not fail

  sebastian-nagel

  committed
  9e41abc

  Copy full SHA for 9e41abc

Commits on Nov 24, 2016

• Match http-equiv meta elements with unquoted attribute values, e.g.

  Show description for 824dd82
  sebastian-nagel

  committed
  824dd82

  Copy full SHA for 824dd82

• HTML encoding detection: fix errors with empty content or empty charset values

  sebastian-nagel

  committed
  607acaa

  Copy full SHA for 607acaa

• Use CharsetDetector to guess encoding of HTML document

  sebastian-nagel

  committed
  5f223d6

  Copy full SHA for 5f223d6

Commits on Sep 21, 2016

• fix: last header was lost if LF LF (intead of CRLF CRLF)

  dportabella

  committed
  9d7abed

  Copy full SHA for 9d7abed

Commits on Aug 7, 2016

• Make regular expression to extract URLs from CSS more restrictive

  Show description for a23cfeb
  sebastian-nagel

  committed
  a23cfeb

  Copy full SHA for a23cfeb

Commits on Jul 13, 2016

• Remove invalid constant

  Show description for 0cbca57
  

  Kristinn Sigurðsson

  authored
  0cbca57

  Copy full SHA for 0cbca57

Commits on May 26, 2016

• Merge pull request #54 from vonrosen/modify-strip-session-params

  Show description for 1485fdd
  

  Kristinn Sigurðsson

  committed
  1485fdd

  Copy full SHA for 1485fdd

• Merge pull request #50 from nlevitt/her-2089

  Show description for cf5f600
  

  Kristinn Sigurðsson

  committed
  cf5f600

  Copy full SHA for cf5f600

• Merge pull request #51 from nlevitt/fix-warc-sizeOnDisk

  Show description for f5e6541
  

  Kristinn Sigurðsson

  committed
  f5e6541

  Copy full SHA for f5e6541

Commits on Mar 22, 2016

• Make canonicalizer be able to strip session id params even if they are the first params in the query string. And add session id strip test. And change IAURLCanonicalizer.java to ensure that if afte…

  Show description for 5cfff50
  

  Hunter Stern

  committed
  5cfff50

  Copy full SHA for 5cfff50

Commits on Mar 11, 2016

• Update ArchiveFileConstants.java

  jrwiebe

  committed
  28c9a1b

  Copy full SHA for 28c9a1b

Commits on Mar 8, 2016

• Store origin-code in ARCRecord header; accessible through getOrigin() method.

  jrwiebe

  committed
  1fede23

  Copy full SHA for 1fede23

Commits on Dec 14, 2015

• flush output etc before tallying stats to fix sizeOnDisk calculation

  nlevitt

  committed
  86589b0

  Copy full SHA for 86589b0

Commits on Oct 28, 2015

• fix for HER-2089 - get rid of broken, seemingly unnecessary escapeWhitespace() step of uri fixup

  nlevitt

  committed
  c1545bc

  Copy full SHA for c1545bc