Talking testing, automation... and anything else.



23
Sep 18

Fail Faster

The primary goal of automated tests is to find failures as quickly as possible. The general idea being that bugs found earlier in development are cheaper to fix. This is generally true, but automation isn’t the only way to fail fast. Here are a few tips for failing faster that have worked well for me, and may work for you too!

Get QA involved in planning

During planning, QA should be building a mental (or even physical) test plan. This is the perfect time to start asking questions about how to test this new feature. Perhaps you’ll need a special resource, like a third-party tool that you’ll use to aid testing, or maybe your tests need to run on an isolated server. Identifying these needs during planning can give you the time you’ll need to acquire them, and get them in place for testing.

This is also a great time to aid testing by baking things right into the app. For example, having a parameter flag in an URL, or an a/b switch, can make a feature much more test-friendly, and impact the speed of your testing effort.

Code review

The importance of meaningful code review cannot be overstated. Whether you’re pair programming or reviewing code before it’s merged, this is a great time to not just find bugs, but to prevent future bugs by gaining a shared understanding of the code, removing complexity and ambiguity, and ensuring code standards are being followed.

Write e2e tests while code is in active development

The best time to write e2e tests is while the dev is actively developing the feature itself. This can be done in a TDD fashion, whereby you create your tests and page objects, have them fail, and get them to pass as the feature is completed. This is also the perfect time to quickly find CSS bugs, and/or add CSS tags to make automating easier. I mean, who doesn’t love a solid ID to grab onto?

Additionally, writing and committing e2e tests directly in the app feature branch can help keep the app code and test code organized until they are both merged… together! It’s also great to have the dev writing the feature, review the e2e tests. Who better to review the tests than the dev that wrote the code! This has the added benefit of keeping devs acquainted with the e2e code.

Handoffs/desk checks

These short meetings are held after the app code is reviewed, but just before moving a story to QA. In this meeting, a dev visually runs through the feature for QA, showing off the feature and answering questions. The primary goal for this meeting is to ensure a shared understanding of the feature, including any changes since it was planned, and any testing tips the dev derived during coding. It’s also a great time to make sure your automated tests (unit/integration/e2e) are up to snuff.

Sluff off on this process at your peril; you WILL regularly find bugs during it. I promise.


06
Feb 15

How To Write A Good Bug

A reader gave mefusca9 flak for writing about automation way more than about testing. Okay. Guilty. So towards making amends for this fact, I give you, Brine’s guide to writing a good bug.

A good bug includes four things:

  • Title
  • Steps
  • Results
  • Impact

How to write a good bug:

1. Write the title last

A title is the most important part of a bug. A good title should convey all (or as much of) the information about the issue… to the point where reading the rest of the bug is unnecessary. And it should do so in as few words as possible.

I used to try to nail the title before writing anything else, but I found it just takes too much time. And I would usually end up rewriting it before submitting anyway. Write it last! I’ve found titles are much easier to write after you’ve written the bug itself.

2. Write concise steps to reproduce the issue

Steps should have just enough detail. Too little and it won’t be reproducible, too much and it can get confusing. If you find yourself writing a bug that has 10+ steps, you either have a very complex bug, or overly complex steps. This can also depend greatly on the bug’s audience and the preferences of the team. Still, keep it concise.

3. Describe the results and the expected results

This is the easiest part. After following your steps, what happened, and what did you expect to happen?

4. Define the impact of the issue:

You might think the impact of your bug is “high” (and what QA wouldn’t), but that term isn’t very useful, and certainly wouldn’t rate a whole section in a bug! No. The Impact section is where you try to PROVE the impact of a bug. This helps the writer because attempting to prove the impact of a bug requires proper regression. It also shows the reader what you’ve tried, and provides insight into the severity of the issue, and areas to verify upon being fixed.

For example:

  • Is there data loss?
  • Is there a workaround?
  • Does it occur on multiple environments?
  • Does it occur in Production
  • Does it occur on multiple browsers/apps/OSs?
  • Did it occur in the last build?

These questions show impact. They tell the reader just how bad, how far reaching, and/or how recoverable, the issue is.

Oh, and there’s one more thing…

4.5 Attach a screenshot

If the issue is cosmetic, it had better have a screenshot attached! Screenshots and/or movies can be worth the entirety of that classic cliché, a thousand words

fin


06
Jan 13

Verify Sorting With Sahi (or any tool really)

4800819674_3cf963deaa_bI recently found a bug when sorting table columns, while running IE9. Ignoring the obvious fix, I wrote up the bug and then as I’m a fan of doing, I wrote a failing automated test to test it (Defect Driven Development!).

Testing sort proved a bit tricky… I thought I would share the results to perhaps save others same pain. I would also not be surprised to find a more elegant solution out there. If you have one, do feel free to share!

My example is in Sahi but it should be easy to transfer to your tool of choice. The gist is:

  1. Sort your column
  2. Collect all the elements in the column in an array
  3. Copy the array and sort the copy using javascript’s sort()
  4. Compare the two arrays

And here’s the Sahi code…

First off, thanks to JavascriptKit.com for providing an example for my example!

The script starts by navigating to JavascriptKit.com and clicking on the Name table header to get our initial sort of that column.

Then we collect each element in the Name column, in table0 and store them in an array, $appSortedValues. To iterate through our loop, we get the number of rows in table0 by counting the number of rows and subtract 1 for each table header.

Now we need a copy of our $appSortedValues array but in Javascript, you can’t just set a new array from our existing array like so: $jsSortedValues = $appSortedValues; . This will actually create a reference to our original array; not what we want. Instead we use the slice() method to select elements 0 through the end of the array and put them all in our new array, thus copying it.

Finally, we use javascript’s sort() method to sort the copy of our array, $jsSortedValues. But the sort() method has a little wrinkle; by default, it sorts alphabetically and is case sensitive. This is unlikely to be how your application’s sort works… but luckily, you can roll your own sort filter by passing a function as an argument to the sort() method. In our case, we want it to be case-insensitive, hence our function caseInsensitiveSort.

Now we have two arrays and can simply use Sahi’s _assertEqual method to verify both arrays are sorted in the same order.

That’s it!


06
Dec 12

Should You Just Stop Tracking Bugs?

With apologies to Ian Betteridge for the hyperbolic headline, I wanted to share this 6 minute lightning talk in which Jon Tørresdal argues for “The simplest solution to bug tracking: don’t!”.

To paraphrase Jon’s list of 5 “crazy ideas”:

  1. Don’t track bugs; just fix them
  2. Delete all bugs in your backlog that you can’t fix immediately
  3. All newly reported bugs are either rejected or fixed immediately
  4. Automated tests are created for each new bug
  5. Set WIP limit for bugs (eg. 20 total)

Ten years ago I would have scoffed at the idea of not tracking bugs and deleting the bug backlog, but today, I can see this as a realistic possibility. In fact, this isn’t far from how my current team operates today. So what’s stopping me from going all in? Guts?


14
Dec 11

Zero Known Bugs!

My current project, internally known as Bug Remediation, had the lofty goal of fixing every single bug for one of my company’s mature products. Today we accomplished just that. One of my apps now has zero known issues. No really… zero!

Now how long that will last is another story…