Things you must do to be successful in a long term software test automation campaign.
1. Don’t give up manual testing.
2. Get team and management “buy-in”
3. Test before you build
4. Identify low-hanging fruit
5. Make re-usable parts
6. Generate your tests, don’t write them. – Automate your automation execution processes.
Don’t give up manual testing:
Automation is great at saving time, but it is terrible at finding bugs. In fact, in order to get your automation working the first time (and get it working enough to trust it) the application usually has to be free of bugs in the area that the automated test is focused. So for new features and new functionality, you almost always need to test manually before you can create and trust automation to test your app (the exception to this is unit tests).
Automation is also not intelligent. A human can be executing a test to validate a user’s ability to create an account, and while testing, can notice a bug that has nothing to do with the test that they started, like maybe dark blue text on a black background which is hard for humans to read. Automating the same test (to verify the ability to create an account) will not result in a usability bug like this being found, unless the creator of the automated script thought to put in a validation for color contrast in the test. But the human doesn’t need to be told these things, human testers just “notice” them as “odd”, and humans can digress to chase down an interesting bug and then come back to their original test.
The moral of this section, don’t think that because you automated your BVT’s that you will catch the same number of bugs you used to catch when you did these manually. There are always undocumented tests that people don’t realize that they are running in their heads and these are hard to capture in automation, until someone catches a bug and then explicitly creates a script to search for the bad condition. Automation will buy you time so that a human can still do the higher level thinking and apply intelligent scrutiny to the application under test, without wasting time to do the mind numbing repetitive tests.
Get team and management buy-in:
To be successful in any automation campaign you need to take the initiative and start automating, but you will not be successful long term if you don’t have buy in from your team and your management.
You don’t need to be a sales person to do this, but it is selling of an idea. And selling is easier with tangible results and an ROI.
Keep a log of how long it takes to do what you do manually and repetitively for a week or month or whatever your test cycle is. Then do some preliminary napkin-math projections in a spreadsheet to see how much time (per month, or per year) you will be spending on doing these activities if you are to continue doing them manually. Build in to your model the expected growth of your test case library and expansion of your regression suite as new features become “old features” and newer features are added.
Once you have that model, follow the rest of the guidelines below to start building and using your automation for your most repetitive tasks. Keep a log of the time it takes to execute your automation, plus the time it takes to maintain your automation through several (feature changing) builds. Use the execution time plus maintenance time per testing cycle and add these into your spreadsheet to make projections side-by-side versus manual. The longer your time projection the more dramatic the savings will be.
Also keep a log of how much time it took to get the initial versions of the tests running for the first time and validated so that you can trust your results. This is the “I” investment part of the “ROI” that you will show in your spreadsheet.
Now depending on your organization you may want to put dollars next to the time savings to show the efficiency you will have created. If you can show efficiency in one area then you can sell the idea of applying the automation model to other areas.
Test before you build …Test Driven Development:
When a team has taken on an automated testing campaign, the tester’s input in the design phase is crucial to the success and timeliness of the development cycle.
Often in a development organization, testing is an afterthought. The test team is something added to the organization after prototyping the first beta application, when there’s not enough time as a developer to complete unit testing and still complete all features on time. But to deliver high quality software products a test team is not only necessary, but their involvement must be sought during the design phase of a product.
Imagine a flight from Seattle to New York departs and is slightly off course at takeoff. If the pilot examines the position early, a slight correction is needed and the error has negligible fuel impact. Early in the flight the distance between the current path and the correct path is small. If the pilot waits 3 hours into the flight to check position and heading, instead of being just a few miles off course now the flight is hundreds of miles off course. The distance between the correct path and the current path has grown over time and it will cost more fuel to traverse the distance back to the proper course. A correction later in the game is more costly.
In any discipline, once a basic foundation of design is established and features are built upon that foundation, changes to the foundation will ripple throughout the product. If the foundation is tested early on for the possibilities it may need to support, then a correction in design is less costly. An error created during the requirements phase can cost 50 to 200 times more to correct later in a project. ,
Developers, Requirements engineers, Business Analysts, Program and Project Managers, should seek the advice of testers during the earliest phases possible of any software development project. Here the tester’s role is to test the ideas presented to see if they will stand up to the unspoken assumptions of the users, and the team building the product.
Testers should be involved with the developers to help define test code to validate the components before they are combined. The tester should work side by side with the developer to write unit tests, and once the unit tests are functional and verified, they should be run before any build of the product is deployed to a test environment.
Testing early saves time.
Identify low-hanging fruit:
Identify in your application the low hanging fruit. These are the tests that are repeated most often. Of those that are repeated most often start with the ones with the least complexity of underlying information (don’t need to read from an excel spreadsheet or have data that becomes stale easily) but the most tedious steps (from a human point of view).
Automation is faster than a human when tasks are repetitive. If you are performing regression testing, then you are taking the same actions you took a day, a week, a month ago (or some measure of your cycle) to validate that existing features, which were not supposed to change; actually did not change. Automation of those actions once may mean that they can be reused for many future "regression testing" cycles. If you have a suite of tests that you run to validate that a build was successful (BVT tests), and this must be run after every build before in-depth functional testing can begin. Automation can increase the speed of execution of these tests, and free up the time that a tester would have spent conducting the same tests manually. The same can be said for “Acceptance tests”, “Regression tests” or “Performance tests”; and in some cases functional tests when the AUT is designed with the proper “automation hooks” in place.
Make re-usable parts:
Once you’ve chosen a starting point and a tool, automate several tests from start to finish and then look for common “action groups” which can be componentized. An example of this common to many web applications is “log in”. If you are testing an application where the user needs to be “logged in” for a variety of the tests, then creating a re-usable module for “log in” is smart. If your application changes – for example if the “login” button changes to a “sign-in” button – it’s much easier to modify one “login” function rather than doing a find and replace in every test that logs in to your application. Also if you know that you will need to log in as sever different user types then perhaps your login function should take a variable like “usertype” or “user name” and “password” as an override. Look for other common things to group – navigation to a certain section, or placing the application in a certain known state are also places where re-usable parts come in handy.
If you are working on a team with others who are automating various features of the same application, then communicate with them. Share your components, and use a source control system. Your whole team can benefit by standing on each other’s shoulders and not re-doing work.
Start small with the most atomic user action groups as functions, then make higher level functions that are groups of the low level functions… such as “login_and_goto_MyAccount(usertype)”. Your tests will become easier to read, and faster to write.
Generate your tests:
Once you have a substantial library of tests and re-usable parts for the “happy path(s)” of your application under test, you may want to start testing the other permutations of those tests. But maintaining several “very similar” versions of the same test can also produce a maintenance headache, especially if your library of tests is growing large. Use a system to “variablize” the common tests into a simple base test that can run with different configurations to accomplish the various validations you seek.
For example, let’s say you have a website that you are testing, and the site is available in several different languages for different countries. In each of the countries some of the features are the same and some might be different due to laws and regulations. You want to verify that what is supposed to be there in each language and country is actually there and you want to verify that what is not supposed to be there isn’t there either. In this situation it is best to have a simple base test that covers the most common path(s), language and country, and then variablize each validation point within the test to allow you to run the test with different “configurations” and still be able to validate what you expect (when what you expect changes with the configuration).
Finally use a tool (or write code) to generate the test instances you need at the time of execution from the base tests and configuration files (or configuration data if you build a database – recommended). Have your tool able to create the language file that your test tool eats and create some method of bootstrapping the automation tool so that you can look for the newly generated test and start your tool running that test… and listen for it’s results to be complete.
We’ve been successful in multiple projects using this approach and have built our own system to generate advanced permutations of our tests from simple base tests and information about the expectations in varying configurations of the application under test. We call our tool TestCentral.
Showing posts with label Software. Show all posts
Showing posts with label Software. Show all posts
Tuesday, November 16, 2010
Thursday, April 9, 2009
Laws as Software of our Nation
How I view the government and laws based on my life's experience as a software tester and an analogy of laws as software:
The basic premise:
Computers answer questions by applying a set of logical rules to an input and calculating the appropriate output or result based on the rule set as currently defined in the software. Often the software takes into consideration the 'state' of the object or the 'state' of the system and bases some of its rules upon this state. Software applications run on top of an operating system which runs on top of hardware and talks to the hardware with a basic input and output system or BIOS.
Our nation runs on Laws. There are laws that constitute our government's basic structure (national and state constitution) which I view as our operating system.
The people of our nation are the hardware. The operating system uses the hardware to determine an output (a jury, a judge, a legislature, etc.) and communicates ideas through them by following the basic operations rules defined within the system. Roberts rules is one example of a BIOS level description of operations allowed within the system. When someone is accused of a crime (a violation of some rule within the system) the information is processed by using a judge and jury to temporarily store the facts of the case in memory and determine an output based on the rule set and the state of the person and the system and to take these operations by following the pre-defined procedures of the courtroom which are based on Robert's rules of order.
If you agree with the above analogy then keep reading.
In software development companies there are various stages of maturity.
A new company may have only a developer as a sole proprietor. The developer's invention or prototype may solve a single problem very well and most of the time is focused on solving a small number of related problems. Often the software starts with a goal or a problem statement in mind and the developer writes code to satisfy the goal or solve the problem. Once they verified that the "happy path" is solved, the software has met its goal and the software is presented to its market.
As software and companies mature they are often asked to solve more and more problems often related tangentially to the original set of problems. As the system gains "weight" or "inertia" they may discover that the solution to one set of problems negatively impacts another prior solution and has unintended consequences. It is at this stage that adolescent software organizations seek to employ "testers" or "software QA engineers".
(readers note: I am a software QA guy so this is where my perspective is applied to our nations laws and our system for creating them etc.)
If you are a software tester and understand the SQA lifecycle you can skip this next section where I define it for those not in our profession:
A software tester (or QA engineer, sometimes also called an analyst) is someone who analyzes and measures a system by conducting experiments, similar to how a scientist conducts experiments and collects data to either prove or disprove a theory. The theory(ies) that the tester is trying to validate stem from the requirements of the system which at some point are based on a goal or problem statement of the system's customer. The system under test is a solution that was created with the goal in mind (the code presented by the developer). If the code passes all of the tests designed by the tester (or if it doesn't pass, but the team determines that the items which failed are an acceptable risk) then the team signs off on the software solution and it is released to its customers as a "new version" of the software.
Most times when a new version software is released, it contains additional features or modifications to the previous version's features. If the tester was involved in previous releases of the software, it is likely that their test plan for the next version will include what we refer to as "regression tests". Regression tests are designed to ensure that any changes to the software don't have a negative impact on previous goals or solutions; that the software doesn't appear to "go backwards" in functionality or "regress" to a less-optimal state; that it doesn't re-introduce problems that the team thought were already solved in previous versions. Any tester worth their salt (and given enough time) will conduct extensive regression tests to verify that future versions of the software don't "piss off" a large group of the software's current user base due to previously solved problems creeping back in, every version of software they release.
Testing takes time. Time is a cost. Not testing has a risk.
Developers are incredibly smart (usually). They have good intentions (usually).
The more solutions contained in a single program, the more likely it is that the program's rules are complex. The better a program handles edge cases (outliers), the more "customized" it's rule set, and the more lines of code exist.
Keeping the rules of a program in your head while writing new rules for the program is difficult, and when the program is sufficiently large, keeping all the existing rules in mind while creating new ones becomes impossible for even the most brilliant humans (perhaps an autistic developer could do this but I've never met one).
Developers don't (usually) intend to create bugs (defects) in software. They are human. When young organizations are bitten for the first time (not just nibbled), is when they start to consider evolving and employing the skills of a tester to help them exercise the existing rules with various scenarios to ensure that:
1. The proposed solution actually solves the problem as intended (the goal is met)
2. Any related pre-existing rules still function and handle things in a manner that is still intended.
3. Nothing "bad" happens (or if it does, it's not so horrendous that users would switch to another product)
If a tester finds a defect - an unintended consequence to a series of events - they usually inform the team by way of a defect report as to the actions taken, the result expected, and the result that actually occurred. The team triages and decides the severity and prioritizes the defect against the other tasks remaining to be completed.
Testing and defect discovery can happen early (while the ideas of solving the problem are still being discussed and formulated) or late (after a solution has already been created).
If defects are found early, all that is required is for the team to change the way they think about the problem and create a better solution.
If the defect is found late (after code is written) it is more costly to fix. Developers have spent time creating code, if that code is thrown away and replaced, all of that time is lost. If the defect is deep, then any logical structures that were based on the underlying structure of rules found to be defective may also require changes. Therefore, the later in a product's lifetime that the defect is found, the more likely that other structures have been created which depend upon the defective structure, and the more likely that "fixing" THAT defect means "fixing" many defects (and subsequent regression testing of more features).
Now for the analogy:
The Constitution is our operating system.
Robert's Rules is our BIOS.
The people are our hardware.
The legislatures (state and national) "congress" are our developers.
The laws they create are the software product (or application) which runs within the operating system by using the hardware.
The executive branch (including police) are the error trapping system, the "Catch" part of a "Try Catch block".
The Judicial system is the "exception handler" and the test team.
- More thoughts on this soon, I need to get some work done -
The basic premise:
Computers answer questions by applying a set of logical rules to an input and calculating the appropriate output or result based on the rule set as currently defined in the software. Often the software takes into consideration the 'state' of the object or the 'state' of the system and bases some of its rules upon this state. Software applications run on top of an operating system which runs on top of hardware and talks to the hardware with a basic input and output system or BIOS.
Our nation runs on Laws. There are laws that constitute our government's basic structure (national and state constitution) which I view as our operating system.
The people of our nation are the hardware. The operating system uses the hardware to determine an output (a jury, a judge, a legislature, etc.) and communicates ideas through them by following the basic operations rules defined within the system. Roberts rules is one example of a BIOS level description of operations allowed within the system. When someone is accused of a crime (a violation of some rule within the system) the information is processed by using a judge and jury to temporarily store the facts of the case in memory and determine an output based on the rule set and the state of the person and the system and to take these operations by following the pre-defined procedures of the courtroom which are based on Robert's rules of order.
If you agree with the above analogy then keep reading.
In software development companies there are various stages of maturity.
A new company may have only a developer as a sole proprietor. The developer's invention or prototype may solve a single problem very well and most of the time is focused on solving a small number of related problems. Often the software starts with a goal or a problem statement in mind and the developer writes code to satisfy the goal or solve the problem. Once they verified that the "happy path" is solved, the software has met its goal and the software is presented to its market.
As software and companies mature they are often asked to solve more and more problems often related tangentially to the original set of problems. As the system gains "weight" or "inertia" they may discover that the solution to one set of problems negatively impacts another prior solution and has unintended consequences. It is at this stage that adolescent software organizations seek to employ "testers" or "software QA engineers".
(readers note: I am a software QA guy so this is where my perspective is applied to our nations laws and our system for creating them etc.)
If you are a software tester and understand the SQA lifecycle you can skip this next section where I define it for those not in our profession:
A software tester (or QA engineer, sometimes also called an analyst) is someone who analyzes and measures a system by conducting experiments, similar to how a scientist conducts experiments and collects data to either prove or disprove a theory. The theory(ies) that the tester is trying to validate stem from the requirements of the system which at some point are based on a goal or problem statement of the system's customer. The system under test is a solution that was created with the goal in mind (the code presented by the developer). If the code passes all of the tests designed by the tester (or if it doesn't pass, but the team determines that the items which failed are an acceptable risk) then the team signs off on the software solution and it is released to its customers as a "new version" of the software.
Most times when a new version software is released, it contains additional features or modifications to the previous version's features. If the tester was involved in previous releases of the software, it is likely that their test plan for the next version will include what we refer to as "regression tests". Regression tests are designed to ensure that any changes to the software don't have a negative impact on previous goals or solutions; that the software doesn't appear to "go backwards" in functionality or "regress" to a less-optimal state; that it doesn't re-introduce problems that the team thought were already solved in previous versions. Any tester worth their salt (and given enough time) will conduct extensive regression tests to verify that future versions of the software don't "piss off" a large group of the software's current user base due to previously solved problems creeping back in, every version of software they release.
Testing takes time. Time is a cost. Not testing has a risk.
Developers are incredibly smart (usually). They have good intentions (usually).
The more solutions contained in a single program, the more likely it is that the program's rules are complex. The better a program handles edge cases (outliers), the more "customized" it's rule set, and the more lines of code exist.
Keeping the rules of a program in your head while writing new rules for the program is difficult, and when the program is sufficiently large, keeping all the existing rules in mind while creating new ones becomes impossible for even the most brilliant humans (perhaps an autistic developer could do this but I've never met one).
Developers don't (usually) intend to create bugs (defects) in software. They are human. When young organizations are bitten for the first time (not just nibbled), is when they start to consider evolving and employing the skills of a tester to help them exercise the existing rules with various scenarios to ensure that:
1. The proposed solution actually solves the problem as intended (the goal is met)
2. Any related pre-existing rules still function and handle things in a manner that is still intended.
3. Nothing "bad" happens (or if it does, it's not so horrendous that users would switch to another product)
If a tester finds a defect - an unintended consequence to a series of events - they usually inform the team by way of a defect report as to the actions taken, the result expected, and the result that actually occurred. The team triages and decides the severity and prioritizes the defect against the other tasks remaining to be completed.
Testing and defect discovery can happen early (while the ideas of solving the problem are still being discussed and formulated) or late (after a solution has already been created).
If defects are found early, all that is required is for the team to change the way they think about the problem and create a better solution.
If the defect is found late (after code is written) it is more costly to fix. Developers have spent time creating code, if that code is thrown away and replaced, all of that time is lost. If the defect is deep, then any logical structures that were based on the underlying structure of rules found to be defective may also require changes. Therefore, the later in a product's lifetime that the defect is found, the more likely that other structures have been created which depend upon the defective structure, and the more likely that "fixing" THAT defect means "fixing" many defects (and subsequent regression testing of more features).
Now for the analogy:
The Constitution is our operating system.
Robert's Rules is our BIOS.
The people are our hardware.
The legislatures (state and national) "congress" are our developers.
The laws they create are the software product (or application) which runs within the operating system by using the hardware.
The executive branch (including police) are the error trapping system, the "Catch" part of a "Try Catch block".
The Judicial system is the "exception handler" and the test team.
- More thoughts on this soon, I need to get some work done -
Tuesday, August 12, 2008
How to become a software tester
How to become a tester…
Someone asked: “What classes should I take to become a tester?” Personally I think you don't really need a class to get in to testing (maybe a coding class to learn the basic structures of software languages). Testing is really a different mindset (but if you must take a class go to www.testingeducation.org which offers a course in black box testing designed by Kaner and Bach).
I’m not sure that good testers are made, I kind of think that good testers are born. They usually have a few skills or traits that are hard to teach.
The characteristics that make a good tester are:
· an inquisitive mind
· attention to detail
· perseverance
· drive
· creativity
· intelligence
· customer focus
· strong personality (be willing to speak your mind when others disagree)
· integrity
· humbleness
· adaptability
· passion for technology
· the ability to learn new things quickly
Also some experience with the scientific method is helpful (any lab science will do), you need to be able to systematically remove things from an equation to find the root cause of failures (bugs), and a science background is helpful.
If you were the kind of kid that took your mom’s vacuum cleaner apart to find out how it worked, then you might have the kind of inquisitive mind necessary to be a great tester.
If you want reading material to learn from the greats, then I'd hit google and look up "Cem Kaner", then look up "Brian Marick", then look up "James Bach", then look up "Bret Pettichord" (there are others I’m sure I’m forgetting but you will find them as you read the previous authors). Read as much of their blogs and white papers as you can. Remember that they aren't always right, (I disagree with them about the benefits of GUI automation, and think it IS possible to manage a functional automation project), but the principals they put forth are dead on.
Then google "TDD" (test driven development) and look for "CTDD" (customer test driven development), then look up "model based testing" to round out what you may hear in the industry. I'd also look at wikipedia for the definition of white box testing.
www.kaner.com
www.pettichord.com
www.satisfice.com
www.exampler.com/testing-com/
Someone asked: “What classes should I take to become a tester?” Personally I think you don't really need a class to get in to testing (maybe a coding class to learn the basic structures of software languages). Testing is really a different mindset (but if you must take a class go to www.testingeducation.org which offers a course in black box testing designed by Kaner and Bach).
I’m not sure that good testers are made, I kind of think that good testers are born. They usually have a few skills or traits that are hard to teach.
The characteristics that make a good tester are:
· an inquisitive mind
· attention to detail
· perseverance
· drive
· creativity
· intelligence
· customer focus
· strong personality (be willing to speak your mind when others disagree)
· integrity
· humbleness
· adaptability
· passion for technology
· the ability to learn new things quickly
Also some experience with the scientific method is helpful (any lab science will do), you need to be able to systematically remove things from an equation to find the root cause of failures (bugs), and a science background is helpful.
If you were the kind of kid that took your mom’s vacuum cleaner apart to find out how it worked, then you might have the kind of inquisitive mind necessary to be a great tester.
If you want reading material to learn from the greats, then I'd hit google and look up "Cem Kaner", then look up "Brian Marick", then look up "James Bach", then look up "Bret Pettichord" (there are others I’m sure I’m forgetting but you will find them as you read the previous authors). Read as much of their blogs and white papers as you can. Remember that they aren't always right, (I disagree with them about the benefits of GUI automation, and think it IS possible to manage a functional automation project), but the principals they put forth are dead on.
Then google "TDD" (test driven development) and look for "CTDD" (customer test driven development), then look up "model based testing" to round out what you may hear in the industry. I'd also look at wikipedia for the definition of white box testing.
www.kaner.com
www.pettichord.com
www.satisfice.com
www.exampler.com/testing-com/
Subscribe to:
Posts (Atom)