Why XOD Switched from Travis to Circle CI

We develop XOD. It’s an open source visual programming IDE available for Windows, macOS, Linux, and browser. From the very beginning, we used Travis CI to build, test, and pack release distros on Linux and macOS. And recently we switched to Circle CI. I’m happy with the change. Why? Read on.

Performance problems

For the last year and a half, XOD has grown significantly in code. And if a year ago the full CI-cycle took 5 minutes or so, recently it went over 25 minutes in machine time. That’s for work-in-progress integration. When we release a new version, the duration can go over 90 minutes.

Put here additional infrastructure problems of Travis itself (more on this below) and merging 3–4 pull requests per day is the maximal pace we can afford.

You can think we have a too massive project that should be split into multiple repos. Well, it is a big monorepo, but it doesn’t stop a developer to entirely build and test XOD locally within few minutes on his machine.

Mac provisioning time

25-minute builds is something we could accept for a while. Few opportunities to parallelize builds still left. But there’s an issue we can’t affect anyhow on Travis. It’s Mac provisioning.

It literally can take hours. That’s no secret. See “Backlog macOS Builds for Open Source projects” on their status page. Hundreds (!) of builds are stacked in the queue waiting for a machine.

The plot shows queue length. The groups of five hills are workdays of a week.

What’s making things worse, I can’t pay to get some priority-pool or something. Two months ago or so they force us to move from commercial plan to the free one just because our GitHub repo is public.

To be fair Travis devs are trying to solve the problem for the last few months. They place some Mac limits per project, etc. But it doesn’t help. I can’t understand why they “simply” can’t double Mac-cluster capacity and take money if you don’t want to sit for ages in the queue.

The last straw

As I have said, when we make a release CI builds the distros. The release process is quite complex per se and involve few hours of careful work. The last thing you want to see is a build failure when you know everything should compile fine because you rehearsed it the day before.

Finally, we just hang. We were not able to release 0.16.0 because functional tests on Travis failed randomly for no apparent reason. Each rebuild cycle took 20 minutes (plus 30–90 minutes for provisioning) and fail again in another “random” point.

A pretty common scenario. The build has not even started after an hour of wait.

Well, there was a flaky functional test problem in our suite. We rely on WebdDriverIO to run the application and simulate user actions. It seems to have a performance-dependent bug which doesn’t appear on local machines and Circle CI as well. We’re going to rewrite the test suite using Puppeteer and headless Chrome, but it will take time, and we want to keep things running until it’s done.

The show-stop caused me to search for a replacement immediately.

Circle CI as an alternative

The choice was pretty easy. I found no CI SaaS which supports Macs other than Travis and Circle. I gave Circle CI a try.

Surprisingly, the migration was quite easy and took less than 24 hours. We were able to release next day.

The switch was simple not because there’re some “import” tools or the configuration structure is very similar, no, they differ much. It’s because Circle CI recently launched their brand new 2.0 platform with excellent docs.

The 2.0 concept is so simple and powerful that I was able to replicate 90% of our CI process with caching, parallelization, GitHub integration, etc in less than a day.

Having few week experience with CircleCI I can compare these two services side by side.

CircleCI vs TravisCI

Performance

Linux regular build: Travis ~12m, Circle ~9m
Linux release build: Travis ~15m, Circle ~9m (no change due parallelization)
Linux provisioning: few seconds on Travis and Circle

macOS regular build: Travis ~21–25m (not stable), Circle ~9m
macOS release build: Travis ~24–45m (not stable), Circle ~9m
macOS provisioning: Travis weekend — few seconds, Travis workday — from 5m to few hours, Circle — few seconds

Branch build wall clock time: Travis — from 30m to few hours, Circle — 9m

Yes, Circle builds XOD in 9 minutes, always, no matter what is the day today and where The Sun is. Travis builds it in 30m if you’re fortunate.

The total time is 09:35. Parallelization helps a lot.

That’s all you should know if you’re going to build on Mac.

Winner: CircleCI

Machines and infrastructure

Travis and Circle seem to use similar Linux-machine configurations. But CircleCI truly supports Docker containers. I can use own docker image to run the build. So I can have everything preinstalled, configured, and tweaked to save few minutes at the build start. On Travis I have to apt-get update && apt-get install over and over again.

Macs are provided by Mac Stadium for Travis. Not sure about their config, but they’re 2× slower to perform the same tasks comparing to their Linux counterparts. I don’t know where CircleCI takes their Macs, but they are equivalent to Linux machines in performance.

Neither of services provides Windows machines. That would be a huge benefit for either one. In that case, it would be the only CI SaaS on the market with all desktop OS’es support.

Winner: CircleCI

Web interface

Travis UI looks old-school. Circle is more modern in aesthetics. But, from the usability point of view, Travis is more clear. They have less visual noise, and more explicit accents done by color and animation. If something is running, it is spinning. If something errored, it is presented in vivid red on a basically gray-scale page.

CircleCI constantly tries to switch me to another GitHub organization which is very confusing and looks like I lost all history.

The dashboard (i.e. the start page) is quite meaningless in CircleCI if you use their modern features and recommended workflow. Buttons to restart a build not always do what you expect from them and the buttons themselves are not always set in intuitive places.

Can you spot the button to retry the failed build?… Yep, there is no such one on the build details page. You have to go back to build list and there will be “Restart” below failed builds. But (!) it will not recover your build anyway 😬

On the other side, CircleCI has desktop notifications. That was the first time when I accepted “Show desktop notifications” from Chrome. They’re very handy for CI.

Both services have room for many UI/UX improvements.

Winner: Travis

Build log

Don’t know what the technical problems are, but on Travis, the log gets damaged from time to time or fails to render at all. The records themselves are pushed to a continuous roll with hard-to-find task boundaries.

On CircleCI I did not found any log problems yet. And the output is clearly separated in collapsible sections with timing information.

Winner: CircleCI

Configuration flexibility

Both services use YAML configuration that you store right in the repo. But architecture is quite different.

Travis defines a pretty rigid pipeline. Your project installs, builds, and deploys. You can deploy here, and there, period. “ — But, but… we want to deploy on Google Cloud Storage. — No way, make it a part of the build step.”

To achieve parallelism on Travis, you should use the Matrix build feature that exists for other purposes. There you assign different sets of environment variables, and then perform a little Bash voodoo to skip some steps of the build and complete other.

CircleCI 2.0 has a workflow concept that’s simple and powerful. It lets you perform jobs fan-out, fan-in, sequences, and all this concerning the branch and tag of Git commit. Super cool.

Both services lack some features. E.g., neither can observe only a part of repo, so I can’t skip lengthy tests if I work on docs. CircleCI configs tend to have much code duplication because I can’t reuse a part of job steps across macOS and Linux. The cache system is pretty basic: in both cases I can’t cache programmatically derived paths, only hard-coded directories.

Winner: CircleCI

Pricing and open source support

Travis charges for workers, i.e. for concurrent jobs. For $129/mo you get 2 concurrent jobs. That’s all, pretty simple.

Circle is a bit more complicated. It charges for Linux and macOS parts separately. You get one Linux worker for free and each additional costs you $50/mo. To use macOS you have to pay $49/mo for 2 workers, but it is additionally limited to 500 minutes of CPU time, so if you have a bigger project, be ready to give $129/mo for 5 worker Macs and 1800 minutes of CPU time.

XOD is an open source project and both services support open source movement (they market their services pretty well, I guess). What does an OSS project get from these services?

Travis lets you build on Linux and macOS for free. $0 for 5 workers. If you’re open source and want to pay for better capacity, you can’t.

CircleCI gives 4 Linux workers for free and 2-worker/500-minutes free macOS. Free Macs available only after a talk with CCI team. Additional resources you can buy on a regular basis.

Winner: Draw

To me the choice is pretty obvious. CircleCI 2.0 seems like a next-generation system comparing with Travis.

But time flows, products evolve. If you’re going to choose a CI SaaS, give both a try first. Each one offers free 14-day trial. I wish both services a good luck.