If you’ve worked with E2E tests for any amount of time, you will have experienced tests that, for whatever reason, randomly fail; aka flakey tests. Such failures can be caused by any number of things: a network glitch, browser barf, app hiccup, act of God, etc…
Test flake can seriously undermine the trust in automation, and drive your automation engineer to drink (or perhaps just drink more). A wise test engineer will embrace that flake happens, and account for it in their automation strategy.
One popular strategy is to re-run failed tests. This way, if a test fails, it gets re-run ‘n‘ number of times, and if it passes, life is good. Enter Protractor-Flake.
Protractor-Flake parses your Protractor test output, looking for failures, and re-runs any failing tests (note: at the spec file level) at the end of your test run. It works really well and is simple to setup. Here’s how…
First install it as a dependency:
npm install protractor-flake --save
Then you can run it directly by using it in place of Protractor (note: the ‘–‘ denotes the end of args passed to protractor-flake, and the beginning of args passed to Protractor):
node_modules/.bin/protractor-flake -- conf.js
Or you can use it programmatically. I like to create a simple node script, which allows me to add logging and reporting (future blog post). Here’s an example:
#!/usr/bin/env node
var protractorFlake = require('protractor-flake');
// skip first two args (node and self)
var protractorArgs = process.argv.splice(2);
protractorFlake({
protractorPath: 'node_modules/.bin/protractor',
maxAttempts: 2,
parser: 'standard',
nodeBin: 'node',
protractorArgs: protractorArgs
}, function(status, output) {
process.exit(status);
});
Saving the above script as a file, 'flake', you could then run your tests thusly:
./flake conf.js --baseUrl=http://qualityshepherd.com
Now obviously, you should be killing off flakey tests to the best of your ability, but Protractor-Flake can save you from non-test-related flakiness, and keep your tests green! You can see a working example in my protractor-example repo on GitHub.
