Add CNAME and robots.txt to the public folder

[obulat]

Problem

Currently, Chooser's SEO score in Google Chrome's Lighthouse report is lowered because we don't have a robots.txt file.
Another problem is that we need to copy CNAME file to the docs folder to make sure that the beta Chooser is deployed properly. Currently, it is done using the npm:build script, but it makes the script unnecessarily long, and there is a better solution.

Description

VueJS copies everything from the public directory into the final build directory. We can use this behavior to add robots.txt and simplify the CNAME file copying. There are four things that we need to do:

1. Create robots.txt file in the public directory with the following content:

User-agent: *
Disallow:

(There is blank space after disallow to allow every page to be crawled)

2. Copy CNAME file from the docs directory to the public directory.

3. Simplify the npm:build script to remove the CNAME copying, as it will be copied automatically by VueJS.

4. Read through the README.md and make changes regarding the build process, if necessary.

Implementation

• I would be interested in implementing this feature.

[Cronus1007]

@obulat Shall I implement this one.
We can implement a bash script to do the same and this will simplify our scripts.

[obulat]

@Cronus1007 I think you should finish the current PR before taking on other issues.

Bash scripts only work on Unix machines, so implementing this in a bash script would make our code not cross-platform, and I strongly oppose that.

The solution I describe in the issue description is much simpler, and it simplifies the scripts.

The npm build scripts will change from:
"vue-cli-service build && shx rm -rf ./docs && shx cp -r ./dist ./docs && shx echo chooser-beta.creativecommons.org > ./docs/CNAME"
to
"vue-cli-service build && shx rm -rf ./docs && shx cp -r ./dist ./docs

[rohan-goel]

@obulat I would like to work on this issue.

[Cronus1007]

@obulat No I am going to implement this issue side by side. So thought of commenting.

Bash scripts only work on Unix machines, so implementing this in a bash script would make our code not cross-platform, and I strongly oppose that.

Yes I agree with this part of yours.

[obulat]

@rohan-goel Sure, go ahead!