Discovery of sites with Hungarian content

[sebastian-nagel]

See the comments in site-discovery-hungarian.sql.

To run the query over multiple crawls:

python src/util/athena_query_multiple_crawls.py \
   --database ccindex \
   --s3_staging_dir s3://mybucket/site-discovery-by-language/hungarian \
   src/sql/examples/cc-index/site-discovery-hungarian.sql CC-MAIN-2018-34 ...

[malteos]

LGTM!

Just out of curiosity:

What's your experience with content_languages like 'hun%' vs like '%hun%'? Does the latter introduce a lot of noise?

[wumpus]

Seb was trying to match the primary language, so the LIKE is anchored on the left side. A 3 language webpage looks like 'hun,fra,eng'. '%hun%' would match the primary, secondary, and tertiary language. Obviously there's more noise in the tertiary side.