Test automation - the solution to all our problems, right? Instead of having all these pesky testers roaming around raising defects where they're not wanted, set up a couple of computers to do the job for us, keep on one tester (just for old time's sake and to turn the machines on in the morning, maybe make the tea as well), big win hours-wise for the project, everyone's happy, yes?
Well, no. Contrary to what seems to be the opinion of many Project Managers, using automated testing is not a "no-brainer". Trouble is, explaining this to them has always been a challenge - until I noticed a real life example everyone will be able to understand that gives an excellent demonstration of the potential pitfalls: the automated ticket barriers at train stations.
I get the train to work every day and over recent months my line has been "upgraded" to have barriers installed, replacing the previous manual inspection by a guard with machines doing it instead. Checking of tickets is effectively a test activity - there is a well-defined specification of the types of tickets that are valid, and only those that match should pass through, the rest being returned to the queue (at the ticket office). It's a very repetitive task so would appear ripe for automation.
At my destination station (which is very busy) there are around 40 barriers available, manned by 10 staff (that's 4 barriers per person, although they need one person on each side so in reality it's more like 8 each) - prior to the barriers there were about 20 staff covering the same area. Note that the addition of barriers does not mean the workforce is redundant, as the barriers need maintenance, there will always be some tickets that don't scan and need manual intervention, and it's also necessary to keep an eye out for rogue cases such as people jumping over or tailgating valid tickets. In terms of a testing project that would be 20 testers reduced to 10, tasked with keeping the automation up to date, catching spurious defects and carrying out exploratory testing looking for new ones.
Fabulous. That's halved the workforce at a stroke, although there are also set-up costs: installation of barriers (creation of test automation framework); construction of tickets that can be scanned (refactoring code to facilitate the automation); and the training of the guards to bring them up to speed with the new technology (learning test automation techniques). But a staff reduction like that is bound to result in a long term saving, so it looks like the sensible approach.
By contrast, my local station is much quieter - before the barriers were installed (just 4 of them) the ticket checking was done by one man (hello Dave if you're reading this) who admittedly wouldn't necessarily catch all of the fare-dodgers but did pretty well. Dave can't be on both sides of the barriers at once so he has a companion (although they have to shout to each other over the barriers). He was away for a week while they trained him up and now he spends all his day letting people through who have the tickets that don't work - my season ticket always works so we don't get much chance for a chat any more, except when I'm queueing to get out (having just two barriers going in each direction has slowed things down a bit). If this were a test project, we would have spent a week's training time in order to be able to double the number of staff required, introduce a bottleneck that slowed down the throughput, and reduce the time available for the tester to interact with the entire project as they are now concentrating solely on automated test failures.
My point? Well, automation is not a good fit for every project, despite what the gurus (and test tool manufacturers) might try to tell you. Aside from the logistics (balancing setup time and increased running costs against test execution time improvements), replacing live testers with automated scripts can lead to a lack of exposure of the remaining team to the actual product as they are stuck in a silo of firefighting test fails; the focus moves from the product to the automation. Real people carrying out real tests can provide a human advantage that may be lacking in an over-automated environment.
There is a place for automation, but it is not a universal panacea for all projects; rather it is another weapon in the tester's armoury, and like all weapons it should be wielded carefully otherwise it can become barrier of its own.
Well, no. Contrary to what seems to be the opinion of many Project Managers, using automated testing is not a "no-brainer". Trouble is, explaining this to them has always been a challenge - until I noticed a real life example everyone will be able to understand that gives an excellent demonstration of the potential pitfalls: the automated ticket barriers at train stations.
I get the train to work every day and over recent months my line has been "upgraded" to have barriers installed, replacing the previous manual inspection by a guard with machines doing it instead. Checking of tickets is effectively a test activity - there is a well-defined specification of the types of tickets that are valid, and only those that match should pass through, the rest being returned to the queue (at the ticket office). It's a very repetitive task so would appear ripe for automation.
At my destination station (which is very busy) there are around 40 barriers available, manned by 10 staff (that's 4 barriers per person, although they need one person on each side so in reality it's more like 8 each) - prior to the barriers there were about 20 staff covering the same area. Note that the addition of barriers does not mean the workforce is redundant, as the barriers need maintenance, there will always be some tickets that don't scan and need manual intervention, and it's also necessary to keep an eye out for rogue cases such as people jumping over or tailgating valid tickets. In terms of a testing project that would be 20 testers reduced to 10, tasked with keeping the automation up to date, catching spurious defects and carrying out exploratory testing looking for new ones.
Fabulous. That's halved the workforce at a stroke, although there are also set-up costs: installation of barriers (creation of test automation framework); construction of tickets that can be scanned (refactoring code to facilitate the automation); and the training of the guards to bring them up to speed with the new technology (learning test automation techniques). But a staff reduction like that is bound to result in a long term saving, so it looks like the sensible approach.
By contrast, my local station is much quieter - before the barriers were installed (just 4 of them) the ticket checking was done by one man (hello Dave if you're reading this) who admittedly wouldn't necessarily catch all of the fare-dodgers but did pretty well. Dave can't be on both sides of the barriers at once so he has a companion (although they have to shout to each other over the barriers). He was away for a week while they trained him up and now he spends all his day letting people through who have the tickets that don't work - my season ticket always works so we don't get much chance for a chat any more, except when I'm queueing to get out (having just two barriers going in each direction has slowed things down a bit). If this were a test project, we would have spent a week's training time in order to be able to double the number of staff required, introduce a bottleneck that slowed down the throughput, and reduce the time available for the tester to interact with the entire project as they are now concentrating solely on automated test failures.
My point? Well, automation is not a good fit for every project, despite what the gurus (and test tool manufacturers) might try to tell you. Aside from the logistics (balancing setup time and increased running costs against test execution time improvements), replacing live testers with automated scripts can lead to a lack of exposure of the remaining team to the actual product as they are stuck in a silo of firefighting test fails; the focus moves from the product to the automation. Real people carrying out real tests can provide a human advantage that may be lacking in an over-automated environment.
There is a place for automation, but it is not a universal panacea for all projects; rather it is another weapon in the tester's armoury, and like all weapons it should be wielded carefully otherwise it can become barrier of its own.
Comments
Post a Comment