Commits

History for

Commits on Aug 29, 2016

Initial Python 3 prep work
sylvinus
committed

Commits on Aug 25, 2016

Exclude nofollow links from the webgraph (#52), rename "datasources" to "dataproviders" to avoid confusion with document sources, and other smaller refactors
sylvinus
committed

Commits on Aug 11, 2016

Smarter defaults for filter plugin + Move partition logic to document sources
sylvinus
committed

Commits on Jul 30, 2016

Convert grep plugin to use Spark SQL
sylvinus
committed

Commits on Jul 29, 2016

Multiple fixes to get PageRank working at scale
sylvinus
committed

Commits on Jul 27, 2016

Upgrade to Spark 2.0.0 and refactor WebGraph + PageRank jobs to use Spark SQL for more performance
sylvinus
committed

Commits on Jul 21, 2016

Commits on Jul 20, 2016

Refactored Spark jobs to use a clean class
sylvinus
committed

Commits on Jul 19, 2016

Scale back on the use of DataFrames because of Java Heap issues
sylvinus
committed

Commits on Jul 18, 2016

Output graph files as Parquet. Move jobs/spark to spark/jobs. Add Hadoop distribution to the local Docker image.
sylvinus
committed

Commits on Jul 15, 2016

Move tree traversal to Cython (3x indexation speedup), reimplement filters as plugins
sylvinus
committed

Commits on Jul 12, 2016

Refactor the indexing pipeline by adding support for plugins & many other small fixes.
sylvinus
committed

Loading