Changing CI provider

[domdomegg]

Background

Travis CI has served us well over the years - I used to use it a lot in my own personal projects. However, travis-ci.org (what we use currently) is being shut down on December 31st 2020. We have to migrate CI provider by then.

We can move to travis-ci.com however they are also altering the rules for open source projects and limiting usage to 1000 build minutes/month. From a quick skim of PRs and master I think our current usage is around 1900 build minutes/month.

Having explored our options, I think GitHub Actions is likely the best fit:

• is much faster (especially at times of load as Travis for open source has a limited fleet of build agents. I did a test tonight timing difference between commit and results: 7 minutes with GitHub Actions, 58 minutes with Travis)
• integrates better with GitHub, not having to have a seperate account and being able to bring checks right into a PR (as opposed to just a status with a link to Travis)
• makes running CI on forks much easier, so it's easier for contributors to use the CI (Travis requires signing up for an account and doing an extra authentication and authorization flow for forks)
• is free for open source with unlimited build minutes
• supports secrets (same as Travis, we can continue publishing alpha play store releases if we choose to fully swap)
• is seeing massively increased uptake because of the above and therefore new contributors are more likely to be familiar with it

I think it's the best option - the other's I've found are:

• travis-ci.com: as mentioned previously, free-tier is limited to 1000 minutes/month which is not enough for us
• Azure Pipelines: free-tier limited to 1800 minutes/month. Very similar to GitHub Actions but less well integrated, more to maintain plus Azure pricing could easily change
• Jenkins: hosted option, could maybe get WMF to do this for us but likely expensive and a nightmare to set up and maintain
• Circle CI and BuildKite: Less integrated with GitHub and not such a well-supported community around it. Also would be worried they may, like Travis, reduce open-source support (less worried about GitHub doing that now they're Microsoft-backed).

Proposal

I propose trialing GitHub Actions for linting and testing alongside Travis to see how it compares for us (i.e. both Travis and GitHub Actions would run).

If we like it, we can add the alpha publishing stuff and remove the Travis build. If we don't like it, we can scrap it and find a better CI provider :)

[misaochan]

Wow, I was unaware that they were shutting down travis-ci.org, and the deadline is pretty close. :/

GitHub Actions does sound like the best option to me, too. We will have to put in the extra work to reimplement alpha auto-publishing, but I reckon that will soon have to be done anyway regardless of what we choose. Pinging @ashishkumar468 @maskaravivek @nicolas-raoul for input.

[nicolas-raoul]

Quoting https://devclass.com/2020/11/25/travis-ci-open-source-engagement/ :

Strictly speaking, this still is in tune with Haase’s promise to “continue to maintain a free, hosted service for open source projects”, however the hoops one has to jump through to get there might mean more projects will reconsider their choices.

Not sure what is the easiest, going through the hoops or switching. I am fine with any choice. A point for Travis is that its software itself is open source, unlike GitHub.

I wonder whether WMF has their own Travis or Jenkins server.

[domdomegg]

A point for Travis is that its software itself is open source, unlike GitHub.

There is https://github.com/actions/runner, which can run an actions workflow and is open source.

[misaochan]

Talked to @ashishkumar468 about this and he is good with GitHub Actions too. I think we can start looking into making this change. @domdomegg are you familiar with setting up GitHub Actions?

[domdomegg]

I've set up a basic workflow for just running the tests in this PR: #4078

We can enable it and run both in parallel for a while to try it out. To get the publishing working from GitHub actions I think we need to update Gradle, which depends on #4090. Do you know who has access to the Google account and can generate these credentials? We will also need to add the decryption key as a secret in the repository settings like we did for Travis.

[misaochan]

If I recall correctly, the WMF Android team set up the credentials for us. It may be difficult to get them to do it again so close to Christmas and New Year. :/ Let's try running in parallel and I will send them an email.

[misaochan]

Hmm, so it seems like our last alpha release was 8 Jan 2021 (after the 31 Dec cut-off), and also CI seems to still work. Any idea if Travis has changed their mind? Or has the cut-off date just been postponed? @domdomegg

[domdomegg]

Hmm, so it seems like our last alpha release was 8 Jan 2021 (after the 31 Dec cut-off), and also CI seems to still work. Any idea if Travis has changed their mind? Or has the cut-off date just been postponed? @domdomegg

I am not sure. My best guess is that Travis have for some reason postponed the cut-off date and haven't communicated that? On their official site it stills says "travis-ci.org will be officially closed down completely no later than December 31st, 2020" but even today we have Travis .org builds running.

I think we should still try to migrate soon to avoid being surprised by the failures, but it is good news that it hasn't but shut down yet!

[madhurgupta10]

@misaochan @ashishkumar468 @domdomegg

I think Github Actions seems to work just fine now, shall we remove Travis altogether?

[misaochan]

Might as well, I don't think Travis has even been running anyway for the past few months.