Talking testing, agility and automation... and anything else.


Re-Run Flakey Tests with Protractor-Flake

If you’ve worked with E2E tests for any amount of time, you will have experienced tests that, for whatever reason, randomly fail; aka flakey tests. Such failures can be caused by any number of things: a network glitch, browser barf, app hiccup, act of God, etc….

Test flake can seriously undermine the trust in automation, and drive your automation engineer to drink (or perhaps just drink more). A wise test engineer will embrace that flake happens, and account for it in their automation strategy.

One popular strategy is to re-run failed tests. This way, if a test fails, it gets re-run ‘n‘ number of times, and if it passes, life is good. Enter Protractor-Flake.

Protractor-Flake parses your Protractor test output, looking for failures, and re-runs any failing tests (note: at the spec file level) at the end of your test run. It works really well and is simple to setup. Here’s how…

First install it as a dependency:

Then you can run it directly by using it in place of Protractor (note: the ‘–‘ denotes the end of args passed to protractor-flake, and the beginning of args passed to Protractor):

Or you can use it programmatically. I like to create a simple node script, which allows me to add logging and reporting (future blog post). Here’s an example:

Saving the above script as a file, 'flake', you could then run your tests thusly:

Now obviously, you should be killing off flakey tests to the best of your ability, but Protractor-Flake can save you from non-test-related flakiness, and keep your tests green! You can see a working example in my protractor-example repo on GitHub.

Plusone Linkedin Facebook Twitter Digg Email

15 comments

  1. Hello,

    Great post. I’m using protractor-flake now and running my tests on Codeship tool, but I noticed that codeship is failing even flake running for the second time and passing all the tests. Do you have any solution so that it can be resolved in codeship?

    • Thanks Rafael! I’m not sure about your codeship issue… we run flake on codeship, so I know it will work. Try your question on StackOverflow. Good luck!

      • (mail bounced from [email protected])

        Hi Brian,

        Good day to you, I’ve found your blog while searching for test automation, wow QA since 1999….. :)

        I’ve just released the beta version of an open-source web automation tool that is based on CasperJS & PhantomJS. Someone shared to Hacker News and the repo stars jumped from 2 to 1000+ over the weekend.

        I’m trying to reach out to experienced QA and test automation folks. May I kindly invite you to take a look if that is something interesting to you or someone you know? I welcome suggestions to make it better!

        https://github.com/tebelorg/TA.Gui

        FEATURES
        – natural language with JavaScript support
        – Chrome extension for recording steps
        – headless (invisible) and visible mode
        – repositories for reusable objects
        – datatables for batch automation
        – auto-wait for element to interact
        – dynamic XPath/CSS element selector
        – run by schedule, command line, REST API
        – support outgoing API calls to webservices
        – cross-platform (macOS/Linux; Windows soon)

        Kind Regards,
        Ken / Tebel.Automation

        • Hey Ken,

          Thanks for the bug report! I just moved my site to a new server, and forgot to add an mx for ‘info’. Fixed.

          Sure, I’ll have a peek. Can’t have too many automation tools! I’m curious about PhantomJS. It’s too much of a pain to use with Protractor, and I just end up testing against chrome, and firefox headlessly anyway (on my own server I call “Otto”, and Codeship CI). Maybe that’s an option with your tool too.

          Thanks for pinging me

  2. I have downloaded and executed the files that you have placed in git for reference.. Execution is successful .

    I have forced one of the ‘it’ to fail . How to rerun the failed test scripts or specs .

    Downloaded and extracted the git zip file and just provided npm update in the project path and started running the file using protractor conf.js. Where the concept flake comes here to help me to rerun.

    I have edited flake file and added path as ‘My project path where conf.js’ is located or I need to give path where protractor is actually installed in my machine

    protractorFlake({
    protractorPath: ‘C:\Users\mythilisri\Downloads\protractor-example-master\protractor-example-master’,
    maxAttempts: 2,
    parser: ‘standard’,
    nodeBin: ‘node’,
    protractorArgs: protractorArgs
    }, (status, output) => {
    process.exit(status);
    });

  3. I intentionally failed one test case by marking ‘expect’ as ‘false’ . This is shown in console as one test case fail out of 15

    Then how would able to run this failed test script

    my project structure is
    — Protractor Example
    —–node-modules
    —–data
    —–pages
    —-Specs
    —-flake
    —-conf.js
    —package.json

    Output:

    C:\Users\XXXXXXXX\Documents\protractor-example-master\protractor-example-master>protractor conf.js
    [07:21:39] I/launcher – Running 1 instances of WebDriver
    [07:21:39] I/direct – Using ChromeDriver directly…
    Spec started

    angular app
    x should add a new friend
    – Expected true to be false.
    √ should delete an existing friend
    √ should not display non-found search terms
    √ should display found search terms
    √ should display no rows when all friends deleted
    √ should display actual count before saving new friend

    non-angular login
    √ should display message for invalid credentials
    √ should display message for empty credentials
    √ should goto friend pages on successful login

    Quality Shepherd blog
    √ should display 5 posts per page
    √ should return search results
    √ unfound search term should return no results
    switching to window 1
    switching to window 0
    A Jasmine spec timed out. Resetting the WebDriver Control Flow.
    x should open social media link in new window
    – Error: Timeout – Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
    √ sidebar should have a set width
    √ should find an older post by paging

    **************************************************
    * Failures *
    **************************************************

    1) angular app should add a new friend
    – Expected true to be false.

    2) Quality Shepherd blog should open social media link in new window
    – Error: Timeout – Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

    Executed 15 of 15 specs (2 FAILED) in 1 min 53 secs.
    [07:23:36] I/launcher – 0 instance(s) of WebDriver still running
    [07:23:36] I/launcher – chrome #01 failed 2 test(s)
    [07:23:36] I/launcher – overall: 2 failed spec(s)
    [07:23:36] E/launcher – Process exited with error code 1

    How can I run my failed test scripts . Please help me on this

    • Hey Sridhar,

      You need to run the tests with protractor-flake, instead of just protractor. Eg.

      ./flake conf.js

      • you mean
        C:\Users\XXXXXXXX\Documents\protractor-example-master\protractor-example-master>protractor-flake conf.js

        or C:\Users\XXXXXXXX\Documents\protractor-example-master\protractor-example-master>./flake conf.js

        both throws error .

  4. Hi Brine,

    Is my way of execution is wrong ?. Please correct me

  5. Hi Brine,

    Working fine > Thanks

  6. All specs are running on failure I need only failed spec to be re-run. Please guide me Brine…..

    • Yeah, this is because protractor-flake parses the stack traces to match failing tests. It’s not grabbing the stack trace from the standard output for some reason. I just pushed some changes to my example code that switch flake to parse "multi" (instead of standard), and protractor to shard tests. This is how I normally run, and it will now only run the failed tests. To get this to work via non-sharded tests, you’d have to fix protractor-flake‘s parse matcher to suit your needs.

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.