Running androidTests in Travis

[domdomegg]

Summary:

I was thinking about how to write a load of espresso tests (see #586) for the app to try to prevent so many bugs cropping up.

To get good coverage, we'd need to be a logged in user. So the tests would be more standardised, I think it would be useful to have a test user the tests could login as.

Therefore I think it would be useful to have an account on beta commons (and maybe even on real/prod commons) which isn't able to do anything - not even edit its own user talk page, or change its own password. Maybe:

• Create the account
• Upload a few test photographs (so the 'home' page is not blank)
• Get it 'locked' somehow

It also might be useful to have a similar test user which is a completely blank account (i.e. no uploads), as I could imagine that causing bugs in the app which we want to test for (#2154).

We could then safely commit its credentials to this repo so people can run the tests. There also might be a different way that doesn't involve sharing credentials, if people have one please do suggest!

The alternate idea I had:

• Require developers to create their own test accounts according to some specific instructions and then
  • Log in to the app in the emulator and then run the test suite
  • Put their credentials in a file that is not commited to the repo (with some .gitignore rule)
• Somehow Travis has its own credentials (kept secret like Google Play keys) and does its testing like that?

The issue with this is I can imagine a lot of new contributors accidentally pushing their credentials to Github. While this would be bad for beta commons, this could be very bad on real/prod commons. The setup would add an extra barrier to entry for developers to start regularly using automated testing, which we want to encourage by making as easy as possible. I also have no idea how to do the bit about Travis.

Would you like to work on the issue?

I don't really know how this would be done - hoping someone else here might. If not, I'm happy to post on the Commons helpdesk and see if we can get some answers.

Yes in terms of trying to add more android tests - although they're so broad and don't really conflict everyone is still free to (and should!) get writing tests. I'm not very good at it yet, but I'm trying to learn :)

[elishaolade]

Hey, how is it going with the tests? I want to help out.

[domdomegg]

Just posted on admin board, hopefully will get a response:

https://commons.wikimedia.org/wiki/Commons:Administrators%27_noticeboard#Support_needed_to_create_a_test_user_with_almost_no_privileges

[domdomegg]

Update: turning this issue into running the tests in Travis. Pinging @maskaravivek cause you're super knowledgeable about Travis + testing!

Also had a thought, need to be careful when running PRs that Travis shouldn't reveal login details e.g. if malicious PR submitted which tried to print the login details. Edit: maybe a possible (but hard to implement) solution would be to have Travis set the login details to a random string each test run and then changing it back to the original secret. Issue is logs are live so there's still the potential for an attacker to login and change it to something else - maybe someone else can take this idea further and develop a solution :)

[maskaravivek]

need to be careful when running PRs that Travis shouldn't reveal login details

Travis allows us to set encrypted environment variables which are not available to Pull requests. These values are available only to the original repo's branches. The only nuance would be that the reviewer will have to run the tests locally to verify if the PR works.

@domdomegg As you pointed out in #2700, we should use mock values for our tests whenever possible. That way we can test the app is isolation. Anyways testing the backend APIs is not our intent. Just for some critical flows, we should use these credentials.

So basically, PRs won't run the connectedCheck task and only master/other branch builds will run this task.

[domdomegg]

Also maybe we could have some way of running the automated tests which don't require login on Travis PR builds - I think the more tests we can run automatically on PRs the better as less likely to break master.

Edit - in a way such that they don't have any access to the environment variables

[maskaravivek]

Are Travis logs visible to all users or is it visible to only members/collaborators? If it is not visible to all users then it should be fine to use non-encrypted environment variables that are available for all builds.

[domdomegg]

Are Travis logs visible to all users or is it visible to only members/collaborators? If it is not visible to all users then it should be fine to use non-encrypted environment variables that are available for all builds.

Travis logs are publicly visible. Also I was thinking more about this and also thought that being really clever someone could submit a PR which has an app which collects the login details, e.g. by changing the login API urls and setting up a sever there.