Commits

Breadcrumbs

History for

onmaster

User selector

All users

All time

Commit History

Commits on Nov 14, 2016

• Merge branch 'master' of github.com:commonsearch/cosr-back

  Show description for 65d71b5
  sylvinus

  committed
  65d71b5

  Copy full SHA for 65d71b5

• Rename "path" parameter to "output" in many plugins. Documentation improvements after feedbacks from PyConFR sprint. Add Python implementation of PageRank for education purposes. Generalize the use…

  Show description for d125c98
  sylvinus

  committed
  d125c98

  Copy full SHA for d125c98

Commits on Nov 7, 2016

• Update to Common Crawl October 2016

  sylvinus

  authored
  1850222

  Copy full SHA for 1850222

Commits on Oct 17, 2016

• Add Commonsearch PageRank Signal (#70)

  Show description for 6b44fee
  

  HenriqueLimas

  authored and

  sylvinus

  committed
  6b44fee

  Copy full SHA for 6b44fee

Commits on Oct 14, 2016

• Fix 'make import_local_data' ETA computation (#67)

  

  AloysAugustin

  authored and

  sylvinus

  committed
  aa80573

  Copy full SHA for aa80573

Commits on Sep 4, 2016

• PageRank fixes for upcoming tutorial

  sylvinus

  committed
  d80d1dc

  Copy full SHA for d80d1dc

Commits on Aug 29, 2016

• Initial Python 3 prep work

  sylvinus

  committed
  a5b06db

  Copy full SHA for a5b06db

Commits on Aug 27, 2016

• Add a test for the Common Crawl source, rename the dump.DocumentMetadata plugin, rename the old "parquet" source to "metadata"

  sylvinus

  committed
  f069527

  Copy full SHA for f069527

Commits on Aug 26, 2016

• Fix lint

  sylvinus

  committed
  3bee4cf

  Copy full SHA for 3bee4cf

Commits on Aug 25, 2016

• Exclude nofollow links from the webgraph (#52), rename "datasources" to "dataproviders" to avoid confusion with document sources, and other smaller refactors

  sylvinus

  committed
  4f0a62a

  Copy full SHA for 4f0a62a

Commits on Aug 13, 2016

• Experimental OAI-PMH source

  sylvinus

  committed
  cc62598

  Copy full SHA for cc62598

Commits on Aug 11, 2016

• Usability improvements to help for the new tutorial

  sylvinus

  committed
  9f0cbe3

  Copy full SHA for 9f0cbe3

• Smarter defaults for filter plugin + Move partition logic to document sources

  sylvinus

  committed
  3d029f2

  Copy full SHA for 3d029f2

Commits on Jul 31, 2016

• Fixes to make PageRank run at scale, first live dump created!

  sylvinus

  committed
  6fe1355

  Copy full SHA for 6fe1355

Commits on Jul 30, 2016

• PageRank improvements, with full set of tests

  sylvinus

  committed
  1419666

  Copy full SHA for 1419666

Commits on Jul 29, 2016

• Multiple fixes to get PageRank working at scale

  sylvinus

  committed
  58c3c9b

  Copy full SHA for 58c3c9b

Commits on Jul 27, 2016

• Upgrade to Spark 2.0.0 and refactor WebGraph + PageRank jobs to use Spark SQL for more performance

  sylvinus

  committed
  e9c1e2b

  Copy full SHA for e9c1e2b

Commits on Jul 21, 2016

• Spark configuration management

  sylvinus

  committed
  872c269

  Copy full SHA for 872c269

• Plugin refactors + split internal & external links

  sylvinus

  committed
  700b3fb

  Copy full SHA for 700b3fb

• Introduce intermediate document metadata Parquet dumps. Misc refactorings

  sylvinus

  committed
  4208e21

  Copy full SHA for 4208e21

• Small runtime fixes for the index job

  sylvinus

  committed
  e860515

  Copy full SHA for e860515

Commits on Jul 20, 2016

• Refactored Spark jobs to use a clean class

  sylvinus

  committed
  01cce4f

  Copy full SHA for 01cce4f

Commits on Jul 19, 2016

• Scale back on the use of DataFrames because of Java Heap issues

  sylvinus

  committed
  373d440

  Copy full SHA for 373d440

Commits on Jul 18, 2016

• Do more validation on collected links

  sylvinus

  committed
  25ba538

  Copy full SHA for 25ba538

• First successful PageRank computation on Common Crawl!

  sylvinus

  committed
  2b540e4

  Copy full SHA for 2b540e4

• Output graph files as Parquet. Move jobs/spark to spark/jobs. Add Hadoop distribution to the local Docker image.

  sylvinus

  committed
  55fde89

  Copy full SHA for 55fde89

Commits on Jul 17, 2016

• Change URL & Domain ID scheme to split PLD & subdomain+path hashes

  sylvinus

  committed
  de4733e

  Copy full SHA for de4733e

Commits on Jul 16, 2016

• Improve static data import & add ETA

  sylvinus

  committed
  2e4db09

  Copy full SHA for 2e4db09

Commits on Jul 15, 2016

• Bypass Elasticsearch bulk helper & fix utf-8 issues with wikidata

  sylvinus

  committed
  ead28c0

  Copy full SHA for ead28c0

• Add wikidata source test

  sylvinus

  committed
  59d84ed

  Copy full SHA for 59d84ed

• Move tree traversal to Cython (3x indexation speedup), reimplement filters as plugins

  sylvinus

  committed
  80bab0d

  Copy full SHA for 80bab0d

Commits on Jul 12, 2016

• Refactor the indexing pipeline by adding support for plugins & many other small fixes.

  sylvinus

  committed
  a3710bc

  Copy full SHA for a3710bc

Commits on Mar 20, 2016

• cosrlib: strip emoji characters

  Sentimentron

  committed
  8cb90bc

  Copy full SHA for 8cb90bc

Commits on Mar 17, 2016

• Fix pylint

  sylvinus

  committed
  73e5175

  Copy full SHA for 73e5175

Commits on Mar 16, 2016

• Ranking signal based on the number of sitelinks in Wikidata

  sylvinus

  committed
  58bd62c

  Copy full SHA for 58bd62c