most of gouv.fr are missing in index/result

[eldk]

URL of the results:

https://uidemo.commonsearch.org/?g=fr&q=gouv.fr

Describe the issue precisely:

Almost all french gov websites are missing.
Some examples :
http://www.nord.gouv.fr/
http://www.prefectures-regions.gouv.fr/

Thanks,

Eric

#2

[sylvinus]

Good catch, thanks @eldk! There are a few separates issues here:

First, searches with domain names should be recognized as such. We could suggest transforming the query in "site:gouv.fr", and maybe give a boost to matching domains anyway. Added as commonsearch/cosr-front#25

Next, nord.gouv.fr is in the index: http://index.commoncrawl.org/CC-MAIN-2015-48-index?url=*.nord.gouv.fr&output=json

However it is indeed hard to find. You have to do:
https://uidemo.commonsearch.org/?g=fr&q=services+etat+nord

The search with "nord gouv" is not working well because "gouv.fr" is in the public suffix list and only "nord" is considered from the domain name:
https://explain.commonsearch.org/?q=gouv+nord&g=fr

Added as commonsearch/cosr-back#31

There is also an issue in the way we score queries with 2 or more words. We seem to return results where one is missing. I'm fixing that immediately.

prefectures-regions.gouv.fr (and others) is not in Common Crawl so #2 is indeed the good issue to link.
http://index.commoncrawl.org/CC-MAIN-2015-48-index?url=*.prefectures-regions.gouv.fr&output=json

thanks a lot and please send more examples :)