Talking testing, automation... and anything else.

Feb 15

Top 10 Automation Blogs of 2014

I am VERY surprised to learn that I made the top 10 in TestBuffet’s top 21 automation blogs of 2014! It’s especially nice to be listed among some of the blogs I read… not to mention discovering some new blogs I have yet to read. Very nice, indeed.

Feb 15

How Much Should I Automate – Part 1

What/how much should I automate in the UI?

blank-stop-sign-clipart-jLcKpqETaEveryone who writes UI automation is faced with this question. At first, it’s often thrown to the side, while the daunting task of getting ANY automation up and running is the initial goal. I can subscribe to this reasonable omission, but eventually, automation’s nemesis will rear it’s nasty little head. Maintenance.

If maintaining ui automation doesn’t eventually become a pain in the ass, you’re either doing it wrong, or smarter than I am (at which point, please leave me the link to your blog where you explain it to us lesser mortals! No… really… do).

And this assumes that you’re a very clever boy/girl, and have already deduced that time is also an issue… and are running your tests in parallel, or in the cloud, or on a grid, or in another dimension (blog link please).

Anyway… there’s no escaping maintenance. So after you’ve refactored, added page objects, and any anything else you could think of to mitigate maintenance, you’re now really ready to start talking about how much you should automate. Sadly, there’s not a lot of conversations out there with real meat on the subject, and/or aren’t generalized into non-usefulness. And probably for the very fact that it takes a lot of doing to even get to this point. Plus, context matters.

How much to automate is a tough question

That said, here’s the answer: SPOILER ALERT: The answer typically ranges somewhere between “it depends” and “everything”.

Yeah… I know… lame. So let me trim that down and say this: in my experience, even in the most robust development environments, and using the best tools available today, automating “everything” in the UI is the wrong answer.

This leaves us with the answer “it depends”… and while this is totally true, I want to try and flesh this out a bit. But I’m out of time for now…

Look for Part 2, coming soon!

Nov 14

Screencast: Protractor Test Automation Framework Example Code

I’ve been playing around with Protractor, a great, new(ish) testing framework from our friends at Google. I’ve shared my example code on GitHub for those that might be interested in such things.

Examples in the code include:

  • Using page objects
  • Running on TravisCI
  • Running tests on Sauce Labs and Browserstack
  • Running multiple browsers at once

I’ve also made a quick screencast that walks you through downloading and running the example code.

Cut to the chase, in the video we:

  • Install Node (okay, I don’t show this but you need do to it!)
  • Download the example code from GitHub
  • run npm install to install the project dependancies
  • Briefly discuss a config file
  • run protractor conf.js to run the tests

Aug 14

Running Geb Tests In Parallel

parallel linesBecause UI tests are inherently slow, running them in parallel–that is, running multiple tests, in multiple browsers concurrently–is all but a necessity.

Luckily, it’s hilariously easy to run Geb tests in parallel, using Maven… all you need do is add the following code to you Maven pom.xml:



And then run your tests via Maven. Eg. mvn test -Dgeb.env=chrome . You can try it for yourself by running my geb-example on GitHub.

This code will spread your tests between up to 4 threads/forks and run tests at the method (test) level. You can also swap out “methods” for “classes” if you’re prefer to run at the spec (class) level. Either way, you can dramatically speed up your automated tests with a few lines of code…

You did think about running tests in parallel when writing your tests… didn’t you?

Jun 14

When To Automate

?There’s a veritable plethora of blogs, articles and papers, listing the mistakes commonly made when creating functional/ui automation… and many of these, themselves, contain common mistakes. Here’s one I recently found…

From an article “Five Common Mistakes and Their Solutions” by Artem Nahornyy:

Starting at the Wrong Time
It is a common mistake to begin test automation development too early, because the benefits almost never justify the losses of efforts for redevelopment of test automation scripts after the functionality of the application changes until the end of iteration…

During the development phase, members of a quality assurance (QA) team should spend more time creating detailed manual test cases suitable for the test automation…

Mr. Nahornyy argues that automating too early can waste efforts due to change, but the same argument can be made against his solution. Changes can also lead to waste when writing detailed manual test cases, early in development.

My advice is to start automating as soon as you can. For example, you might write automation code when devs are writing app code. Start with a failing test, and flesh out any page objects needed (or create them if they don’t exist). Then create any methods and/or data needed by the tests. This will save time later for manual/exploratory testing, when the code is complete.

Even better is if you can get access to the code in progress. Maybe by accessing your dev’s machine, or working from a branch, etc… Then you can start hooking up selectors in your page objects and perhaps even running code. More importantly, you can sniff out early bugs in the html/css/javascript, or even in the ui. This is also the perfect time to ask for css changes to solidify page object selectors. Eg. add a css id to an element to save you from using a more brittle solution. It’s WAY easier to get a dev to add an id while they’re still in the code, rather than later down the line.

Truth is, I would generally argue that it’s a mistake to automate too late, rather than too early. The ability to run even the simplest automated tests during a release can add real value. Kicking the can down the road with automation might at best, bloat your backlog, and at worst, never come to be.