Why is it so challenging to get users involved in User Acceptance Testing (UAT)? Isn’t it called UAT because the users are the main participants? My experience has shown that involving users in all phases of the project, especially UAT, is the best way to ensure project success. This article will present a proven approach to increasing user involvement by addressing the problems with traditional approaches to UAT.
I recently worked on a project in which a major defect was found after the software application moved to production. This defect caused the users to perform three days of manual processes. Users on the IT project team worked countless overtime hours. The defect also resulted in a frustrated user group and business sponsor. The project team’s morale was low and the business users lost a great deal of confidence in the project team’s ability to deliver quality software solutions. To reduce the risk of making this crucial mistake in the future the project team improved the UAT approach by getting users more involved.
Traditional Approach
Too often User Acceptance Testing is not taken seriously. For many reasons UAT gets shortened, is not conducted in a way that ensures a successful project, or the worst scenario, is not done at all.
An approach I have used in the past consisted of the project team members – Business Analysts and QA Analysts – writing test scripts and providing demos of the new application to the users. The users would then walk through test scripts step-by-step. In some organizations the BAs write and execute the UAT tests and then present the results to the users for sign-off.
Traditional approaches are often not effective because they are missing a key ingredient—the right amount of user involvement. In the project previously mentioned, there were five major issues relating to UAT that we had to address. These are common problems in many organizations related to lack of user involvement:
- Users may not be fully vested in UAT. In the traditional approach the users are directed by the BA during UAT and are brought in too late in the project to have an impact on the test plans. This results in a lack of ownership by the users and less responsibility on their part for the success or failure of the project.
- Users do not fully understand how the new functions should work when they are asked to test them. Just seeing a demo istypically not enough. This can result inthe UAT session becoming a trainingopportunity and not a true test.
- Tests are often generic and are not all based on real-life scenarios. If the testscripts are written by the IT projectteam, there is a greater risk for missingreal-life scenarios. This is because theIT project team does not use the application every day like the user.
- Project team members are usually pressed for time. Often a BA has already beenassigned to perform requirementsactivities on another project duringUAT. Balancing multiple projectsmeans that BAs have a hard timefocusing on UAT, while meeting theirother project deadlines.
- High pass rate of UAT test plans. Ironically, this is not a positive thing.Often a BA writes test scripts and teststhem himself prior to UAT to ensurethe scripts pass. When a BA writes thetest scripts the users are not given anopportunity to interject enough real lifescenarios to validate the system.
A Recommended Approach
To address these common issues and increase the chance for project success we need to take a new approach to UAT.
1. Involve key users early
Once Quality Assurance (QA) begins testing, UAT planning should start. Identify users who have a deep understanding of the business requirements and are change agents for the group. Identify all of the tasks that need to be accomplished, the owners of each task, and a high-level timeline. Doing this will help determine if all the right people are involved.
2. Provide hands-on training of the system for the UAT participants
Providing a demo is not good enough. Once QA feels the application is stable enough, give hands-on training to the UAT participants. It is critical to explain to the users that issues may arise because QA testing is not complete. Ask the users to stay focused on how the application works and not so much that it is not fully operational.
3. Use facilitation sessions to create test plans
Have the users write their own test plans. This may sound far fetched, but it is keyto getting UAT as close to real life aspossible. The BA’s primary role is tofacilitate the UAT test plan creationprocess, but not to write a single testscript. Using process workflow diagramsand Use Case documentation from therequirements package, ask the users todetermine what processes and systemfunctions need to be tested. Provide theUAT participants with examples of testscripts and explain the need to capturethe goal of each test, the necessary steps,and the expected results. The stepsbecome second nature to the users of thesystem and they find it difficult todocument each step they take to accomplish a goal. Help them think through their processes in detail to ensure they have documented each taskcompletely.
Review the test plan. Once the test plans are written, the BA reviews the test plans to ensure all the necessary functions and processes impacted will be tested.
Determine necessary inputs and outputs. Once all of the test plans are written, askthe users to document the inputs theyneed to complete each of their test scriptsand the outputs that will be generated. Make sure all UAT participants have thenecessary inputs to complete their testsbased on all of the outputs. If some aremissing, enlist other users to create thoseinputs during testing execution.
Make it as close to real life as possible. To enhance the real life feel, the BAs work with the users to determine a testing schedule. Make sure the schedule follows their daily process. Again, use process workflow and Use Case documentation to ensure the test plans are executed in the order the activities would be done in real life.
4. Ensure users execute the test
The BA’s role is to ensure the test environment is set up and to assist the users as they execute the tests. The user’s role is to execute their tests and document the results.
The Results
Using this approach can help reduce the risk of major defects making it to production and ensures the users are satisfied that the solution meets the objectives of each project. Here are some of the key results from using the improved approach:
- The UAT participants take responsibility for the success of the release. They feel part of the team due to the collaboration with the IT project team and involvement in UAT planning and test creation and execution. They also help champion the benefits of each release to the larger user base.
- Due to the pre-test training, users are comfortable with new application. This allows the users to develop real life test scenarios and the time allotted for testing is not used for training.
- Since the users create and execute test plans, the tests are very close to real life scenarios and the users are more comfortable running the tests.
In addition there are tangible benefits to future projects:
- QA incorporates the scenarios documented by users in the QA test plans for future releases.
- Over time there is decreased use of the BA’s time for UAT. With the BA facilitating the UAT process and not doing most of the work, the BA can focus on other necessary tasks like launching the next project.
Implementing the Approach
A lack of user involvement in UAT is notuncommon. I urge you to try thisapproach even if you have notexperienced a drastic wake-up call such asmajor defects in production. As we arecalled upon to deliver solutions faster andfaster, it is just a matter of time beforemajor defects make it to production.Here are some tips for getting started.
- Start small. To help manage the changes with the new approach, identify a release with a low number of users and/or new functions. This will allow you to test the new process, discover lessons learned, and make the necessary adjustments.
- Plan for additional time. Using this new approach will initially require more time. Work with your Project Manager to plan more time into the UAT phase for your first 2-3 projects. As you get accustomed to this approach, it will require less time.
- Identify power users and champions of application. They are your besttesters and have the most interest inthe project’s success.
- Sell the benefits of the new approach to your users. As with any newapproach, BAs need to help the usersunderstand what the approach is andhow it will ultimately improve theirbusiness.
- Save the user test plans for future releases. Reuse of test plans will helpspeed up the time dedicated to UAT infuture releases and can be used toupdate your QA test plans.Successful implementation of thisapproach helps ensure projects meet theuser needs. The collaboration of userswith the project team leads to a sharedresponsibility for the success and failureof the project.
Author: Jonathan “Kupe” Kupersmith, BA Certified, CBAP®
Director of Client Solutions, B2T Training