Commits

Breadcrumbs

History for

onmaster

User selector

All users

All time

Commit History

Commits on Oct 6, 2022

• Improve anchor text extraction

  Show description for b2be0a5
  sebastian-nagel

  committed
  b2be0a5

  Copy full SHA for b2be0a5

• Improved text and anchor text extraction

  Show description for 91e7052
  sebastian-nagel

  committed
  91e7052

  Copy full SHA for 91e7052

Commits on Oct 5, 2022

• Use WARC header WARC-Identified-Payload-Type (if available) to identify

  Show description for b42ecb9
  sebastian-nagel

  committed
  b42ecb9

  Copy full SHA for b42ecb9

Commits on Jun 15, 2020

• WAT extractor: do not fail on missing WARC-Filename in warcinfo record, fixes #88

  Show description for d2c7379
  sebastian-nagel

  committed
  d2c7379

  Copy full SHA for d2c7379

Commits on Jun 5, 2020

• WET extractor: add identified natural language of text content (#22)

  Show description for a1f07ad
  sebastian-nagel

  committed
  a1f07ad

  Copy full SHA for a1f07ad

• - simplify date parsing resp. do not parse if not needed

  Show description for 37b40f5
  sebastian-nagel

  committed
  37b40f5

  Copy full SHA for 37b40f5

Commits on Dec 12, 2019

• Work-around issue processing WARC request records with invalid message

  Show description for 428022b
  sebastian-nagel

  committed
  428022b

  Copy full SHA for 428022b

Commits on Aug 28, 2019

• WAT: only unescape complete XML/HTML character entities (fixes #19)

  Show description for bb754a3
  sebastian-nagel

  committed
  bb754a3

  Copy full SHA for bb754a3

Commits on Aug 27, 2019

• WAT: only unescape complete XML/HTML character entities (fixes #19)

  Show description for 716eae9
  sebastian-nagel

  committed
  716eae9

  Copy full SHA for 716eae9

Commits on Aug 19, 2019

• WAT: only unescape complete XML/HTML character entities (fixes #19)

  Show description for 60b6f7e
  sebastian-nagel

  committed
  60b6f7e

  Copy full SHA for 60b6f7e

Commits on Jul 23, 2019

• Use openjson as replacement for the org.json dependency

  Show description for d188b9e
  sebastian-nagel

  committed
  d188b9e

  Copy full SHA for d188b9e

Commits on Jul 22, 2019

• Replace the org.json dependency by Ted Dunning's open-json library

  Show description for 470c353
  sebastian-nagel

  committed
  470c353

  Copy full SHA for 470c353

Commits on Jul 8, 2019

• WAT/WET extraction: use compiled regex Pattern to get better performance

  sebastian-nagel

  committed
  4a1f323

  Copy full SHA for 4a1f323

• WAT: unescape XML/HTML character entities (#14)

  Show description for d5af4e5
  sebastian-nagel

  committed
  d5af4e5

  Copy full SHA for d5af4e5

Commits on Aug 23, 2017

• WAT extractor: get links from onClick attributes, fixes #8

  Show description for ea0729b
  sebastian-nagel

  committed
  ea0729b

  Copy full SHA for ea0729b

• WAT extracor: add rel attribute to A@/href links, fixes #10

  Show description for dda2c89
  sebastian-nagel

  committed
  dda2c89

  Copy full SHA for dda2c89

• WET extractor: improve spacing of textual content, fixes #13

  Show description for 7ce4e88
  sebastian-nagel

  committed
  7ce4e88

  Copy full SHA for 7ce4e88

Commits on Jun 5, 2017

• Limit pattern matching URLs embedded in CSS to match max. 8000 characters,

  Show description for ea00e6d
  sebastian-nagel

  committed
  ea00e6d

  Copy full SHA for ea00e6d

Commits on May 4, 2017

• Fix HTTP-Response-Metadata for wget WARCs. Changes came from https://github.com/commoncrawl/ia-web-commons/commit/58e85a60d75707da55ed499f836e57d49347484a

  

  ldko

  authored and

  sebastian-nagel

  committed
  3fac189

  Copy full SHA for 3fac189

Commits on Apr 28, 2017

• Do not add value of preceding HTTP header field if there is no value (or

  Show description for aad4e05
  sebastian-nagel

  committed
  aad4e05

  Copy full SHA for aad4e05

• Merge with recent master of iipc/webarchive-commons, transfer all

  Show description for 58e85a6
  sebastian-nagel

  committed
  58e85a6

  Copy full SHA for 58e85a6

Commits on Feb 22, 2017

• Improve HTML link extraction

  Show description for 11579c2
  sebastian-nagel

  committed
  11579c2

  Copy full SHA for 11579c2

Commits on Feb 15, 2017

• Merge pull request #69 from sebastian-nagel/cc-url-parser-ignore-empty-port

  Show description for cb91063
  ldko

  authored
  cb91063

  Copy full SHA for cb91063

• Merge pull request #68 from sebastian-nagel/HTMLResourceFactory_to_use_CharsetDetector

  Show description for ed33bae
  ldko

  authored
  ed33bae

  Copy full SHA for ed33bae

• Merge pull request #65 from dportabella/fix

  Show description for 1ee2529
  ldko

  authored
  1ee2529

  Copy full SHA for 1ee2529

• Merge pull request #62 from iipc/kris-sigur-patch-1

  Show description for c5876b3
  ldko

  authored
  c5876b3

  Copy full SHA for c5876b3

Commits on Jan 27, 2017

• CharsetDetector: remove unnecessary check for empty string (contributed by @ldko)

  sebastian-nagel

  committed
  0384028

  Copy full SHA for 0384028

Commits on Jan 18, 2017

• CSS link extraction: clip also unpaired leading and trailing quotation marks

  sebastian-nagel

  committed
  194a1fa

  Copy full SHA for 194a1fa

Commits on Jan 4, 2017

• Improve clipping of quotation marks in CSS link extraction

  Show description for b918f7f
  sebastian-nagel

  committed
  b918f7f

  Copy full SHA for b918f7f

Commits on Dec 9, 2016

• Strip empty port, do not fail

  sebastian-nagel

  committed
  9e41abc

  Copy full SHA for 9e41abc

Commits on Nov 24, 2016

• Match http-equiv meta elements with unquoted attribute values, e.g.

  Show description for 824dd82
  sebastian-nagel

  committed
  824dd82

  Copy full SHA for 824dd82

• HTML encoding detection: fix errors with empty content or empty charset values

  sebastian-nagel

  committed
  607acaa

  Copy full SHA for 607acaa

• Use CharsetDetector to guess encoding of HTML document

  sebastian-nagel

  committed
  5f223d6

  Copy full SHA for 5f223d6

Commits on Sep 21, 2016

• fix: last header was lost if LF LF (intead of CRLF CRLF)

  dportabella

  committed
  9d7abed

  Copy full SHA for 9d7abed

Commits on Aug 7, 2016

• Make regular expression to extract URLs from CSS more restrictive

  Show description for a23cfeb
  sebastian-nagel

  committed
  a23cfeb

  Copy full SHA for a23cfeb