We can automate testing actions:

Some tasks that are easy for people are hard for computers. Probably the hardest part of automation is interpreting test results. For GUI software, it is very hard to automatically notice all categories of significant problems while ignoring the insignificant problems.

The problem of automatability is compounded by the high degree of uncertainty and change in a typical innovative software project. In market driven software projects it's common to use an incremental development approach, which pretty much guarantees that the product will change, in fundamental ways, until quite late in the project. This fact, coupled with the typical absence of
complete and accurate product specifications, make automation development something like driving through a trackless forest in the family sedan: you can do it, but you'll have to go slow, you'll do a lot of backtracking, and you might get stuck.

Even if we have a particular sequence of operations that can in principle be automated, we can only do so if we have an appropriate tool for the job. Information about tools is hard to come by, though, and the most critical aspects of a regression test tool are impossible to evaluate unless we create or review an industrial size test suite using the tool. Here are some of the factors to consider when selecting a test tool. Notice how many of them could never be evaluated just by perusing the users manual or watching a trade show demo:

♦ Capability: Does the tool have all the critical features we need, especially in the area of test result validation and test suite management?

♦ Reliability: Does the tool work for long periods without failure, or is it full of bugs? Many test tools are developed by small companies that do a poor job of testing them.

♦ Capacity: Beyond the toy examples and demos, does the tool work without failure in an industrial environment? Can it handle large scale test suites that run for hours or days and involve thousands of scripts?

♦ Learnability: Can the tool be mastered in a short time? Are there training classes or books available to aid that process?

♦ Operability: Are the features of the tool cumbersome to use, or prone to user error?

♦ Performance: Is the tool quick enough to allow a substantial savings in test development and execution time versus hand testing.

♦ Compatibility: Does the tool work with the particular technology that we need to test?

♦ Non-Intrusiveness: How well does the tool simulate an actual user? Is the behavior of the software under test the same with automation as without?