Import DNS metadata

[sylvinus]

There are a few ranking signals we could extract from data related to the domain name registration and records:

• Date created, expiry date, date of last owner change?
• IP (to be used later to fight spam?)

Getting complete whois/zonefile dumps doesn't seem easy at the moment. Any ideas?

A couple interesting links:

• https://news.ycombinator.com/item?id=10367342
• https://scans.io/study/sonar.fdns

[sylvinus]

Greg mentioned that Internet Archive does "survey crawls" every year with some root zones, but they are pretty spammy, as expected.

[thepieuvre]

Why not starting by indexing only domains from indexed Web pages and doing update when pages are updated? it is not complete, but it is a start.

[sylvinus]

@thepieuvre currently we reindex everything each month so we can avoid doing updates

However making sure we have the root domain indexed for each page is a good start indeed.

[indolering]

Those spammy URLs are just as valuable as non-spammy URLs.

[sylvinus]

@indolering yes there are as a dataset, but we don't want to have them in the search results, the index will be large enough as it is!

[indolering]

But wouldn't it help to know which pages those spammy domains link to?