How We Handle Quality Assurance (QA) in Stormotion

*“The bitterness of poor quality remains long after the sweetness of meeting the schedule has been forgotten”, — Karl Wiegers, an American software engineer*

We believe that this quote hits home for almost every competent IT specialist, and Stormotion is not an exception at all. Obviously, it’s impossible to achieve high quality without testing the product and that’s where Quality Assurance comes to help.

The clients are usually also interested in proper testing of their product. In general, QA Engineer finds and confirms around 80-90 bugs for a project of medium complexity. In bigger projects it’s rational to expect more than a hundred bugs appearing. Imagine releasing a product of such quality, keeping in mind that there are also smaller bugs that aren’t making it to documentation. We don’t think that such a project is going to fulfill the customer's business needs, so Quality Assurance is without a doubt vital for success.

In general, the bugs found throughout the testing are not the result of developers being incompetent or them making wrong decisions. Usually, they appear in situations that can’t be predicted and are only seen after implementation. For example, it may turn out that some requirements are absolutely incompatible in a single project, or the level of required customization supposes configurational changes that may lead to unforeseen results. Also, we can’t forget about the bugs caused by the appearing technical debts and the team’s ways to handle them.

If you are interested in Stormotion’s way of solving technical debt issues and some general tips from our experts, please make sure to check the article down below.

Project Manager. In our team, the QA cycle is a so-called safety net. In case our developers are unable to conduct Smoke Testing, our PM is the one to provide the crash-free build — code pieces compiled into a program which is executable yet remains unreleased — to the QA engineer. Also after our QA specialist fills in the documentation, the PM always needs to verify it.

**Designers. Even though their role isn’t directly associated with the QA cycle, the design they create needs to be maximally convenient for the user. **To achieve the best UX, they should conduct usability testing in the process of creation, before delivering the app to the QA.

QA Engineer. The person whose role is to make sure that the delivered software build is of the highest quality. In brief, their responsibilities include:

• Verifying the filling of the Product Backlog according to its rules and canons.
• Making sure the UX/UI parts are convenient and ergonomic without any unfinished actions and features.
• Ensuring every version is delivered on time along with creating a report for every Testing Round for the Tech Team to see the bigger picture.
• After the build release, conduct Sanity Testing to proceed to the next QA stage.

As Norsk Guardian is a project more inclined to manual testing as some main features such as Bluetooth can’t be automated. Thus, after discussing with the team, we decided to save time by sticking up to manual testing instead of an automated one.

Common activities for the Requirement Analysis stage are:

• Identification of testing types to be performed. The QA engineer should discover the requirements and decide which types of tests will be performed.
• Verification of the backlog structure and user stories. Software requirements are usually expressed as user stories. The QA engineer should verify the backlog structure and avoid such issues as the UI design not matching the context of the user story. Also, the QA engineer should verify that every user story is uniquely identified, complete, traceable, testable, and prioritized.

One of the Norsk user stories stated that users should be offered to set up a notification reaching a custom charge level. When verifying the backlog and user stories, the question was where to locate the required notification block in the UI.

A "Set Custom Charging Level Notification" button should be found after proceeding to the "More Options" section in client’s "Batteries Group". Its location between the “Delete Boat” and the “Rename Boat” features is necessary in order to perform regression tests in a row. A QA engineer doesn't need to move to another section and spend extra time on these actions.

• Verification of UX/UI design. At this stage, the QA engineer verifies the conformity of UX/UI, performing Usability testing to avoid illogical patterns. Also, this is when accessibility testing can be performed.
• Identification of test environment details where testing is supposed to be carried out. The Quality Assurance engineer should prepare templates for the main documents which will be used on the project (Test plan, Traceability matrix, Checklist, Test cases docs).
• **Automation feasibility analysis. **The necessity of automation testing implementation should be directly discussed with the team. On some projects, automation testing shouldn't be required.

 
 

Test Planning

Test Planning is a stage for QA specialists to determine the strategy for a test plan, testing schedule, cost estimates, test environment, and limitations. All the risks should be also determined as well as The Test Plan should be finished by the end of this phase.

Common activities for the Test Planning stage are:

• Preparation of test plan/strategy document for various types of testing. The QA engineer should analyze the requirements and then decide which type of testing will be used to deliver a high-quality project.
• Testing tool selection. This stage supposes a QA Engineer to determine all the tools they are going to use in the process.

Among the testing tools for Norsk Guardian our QA Engineer used Slack, Google Sheets, Active Collab, TestFlight, Figma, Android Studio, and others.

• Test effort estimation. At this stage, QA specialists with the PM themselves create an estimate containing the task, the ones responsible for it, and the required number of hours.

On the Norsk Guardian project, we identified the main functionality, which we would cover through tests. Sometimes, in the case of a low functionality project, we would cover by tests 70-80% of all functions, but on large projects, the test cases cover only 20-30%.

 
 

Test Case Development

Test Case Development stage is about creating new and modifying already existing test cases and scripts after finishing the Test Plan.

Common Activities for Test Case Development:

• **Test case creation. **The structure of test cases at Stormotion is described in the Master Test Plan. When the "test case development" step is completed, we start working on a traceability matrix.
• **Reviewing test cases. **The Test Cases should be reviewed by another QA engineer to verify that they include everything from the functionality.
• Traceability matrix creation. Requirement Traceability Matrix (RTM) — is a document used to map and track the test cases representing user requirements.

 
 

Test Execution

Test Execution supposes exactly the process of test script implementation and bug reporting after the test plan is finished. The bugs reported to a development team are getting fixed for the further iteration of test execution.

Common activities for Test Cycle Closure are:

• Executing tests after the test plan. While filling out the Test Plan document our QA should decide when and what types of testing to use on the current project. Therefore, during the execution, our specialists should stick to the plan they've chosen before, unless there are required changes from the customer. In QA plannings customer’s changes also suppose changes in testing documentation.

On the Norsk Guardian project, our QA decided to include localization testing as the project will focus not only on the English-speaking market.

• Retesting the fixed build. As soon as the updated build reaches our QA Engineer, they are performing a retest on the fixed build.
• Reporting the status of each iteration. The QA engineer should create a report after every iteration of Testing. After that, they perform a feature or regression testing as soon as possible to give our developers time to fix some issues before the Demo.

For convenience, the document for QA logs was created to track the time of verifying the version and understanding whether the build is stable. The report should include the build version, a number of issues, fixed bugs, and the list of “Open” bugs.

• Manage the project's status in the QA logs doc. At Stormotion we have a document for QA logs which allows us to track the status of all projects. This document was created to help our development team to understand whether the version is stable as well as to structure the development phases.

 
 

Test Cycle Closure

Test cycle closure is a phase where obviously the testing cycle and its execution are coming to an end. With filling the test completion reports, this is when the QA team gathers and analyzes information regarding the processes of this project. After the discussion, it becomes clear what aspects weren’t as good as they could be and what to do in the next projects to enhance efficiency.

Here are some tips from our team on how to achieve a maximum of QA productivity:

• Be actively involved in the development. Spend a little more time launching the product yourself, and don’t only rely on demos. Our team provides project guides and tests every single version to make it convenient for our customers to even test it themselves.
• Be available for the team as much as possible. In case you have additional questions or updated requirements it’s always better to provide them immediately. And vice versa, whenever the Team requires extra clarification, you are able to make the processes more efficient by being available more often.
• Determine your precise requirements for the future project and try to avoid changing them too often.
• Don’t be afraid to ask more questions and discuss everything regarding bugs and QA right away.
• Thoroughly acquaint yourself with the project documentation. Our team always provides a Test Plan and a list of potential risks for the client to be aware of the processes and the testing order.
• In case the project supposes connecting several different devices and we are unable to have them all in our office, we recommend launching new tested versions on them as well.

 
 

❓ What to do if a customer has their own QA Engineer on the Team?

There are cases when customers already have the QA engineer in their team and are willing to take control of the process. Generally, we see three potential scenarios of how it can be done:

 
 

QA is strictly conducted by our specialists

This is the most common scenario from our experience. Usually, a client doesn’t have any objection to letting our QA engineer do their job. We highly recommend entrusting the Quality Assurance to the team responsible for the development, as it is more convenient in such ways:

• Greater communication levels. When there’s a strong link between QA, developers, designers, and a PM, it drastically enhances the productivity of the Quality Assurance stage. Communication between the team members is built up by the constant back-to-back project development and can’t even be compared with the other scenarios in terms of efficiency.
• Matching time zones. There will be no trouble in additional handling time zone differences among the team members.
• Unique approach. Every Tech Team has its own boundaries, visions, and approaches to a project. The most ineffective thing to do in such an environment is to trust an outsider expecting him to enhance productivity. It often becomes quite the opposite, taking more time to adapt and understand the approach and priorities which the Tech Team had been working on from the beginning. Thus, making the QA stage much longer and more expensive.
• No language barriers. There’s also no need to seek convenient ways for communication among the team members in terms of a preferred language.

 
 

QA is conducted by the client’s specialists

This scenario is certainly less common and the risks it brings compared with the first case clearly show why. Letting a customer’s side to conduct a Quality Assurance stage usually implies such risks:

• Chances of exceeding the estimates are significantly higher.
• Required additional time for a specialist to adapt to the development team and its workflow.
• Possible time zone differences, which creates inconvenience both in communication and development.
• The team must speak the language of the specialist provided by the customer, playing a crucial part in making communication and adaptation even harder.

 
 

Compromised QA conduction

It may sound like nonsense, however, unlike the previous one, this type of QA stage even has a way to be efficient. This case may be used in larger projects, where the scope of work is noticeably bigger. With the compromised QA conduction on both sides of the development, it becomes possible to find bugs and problems way faster.

However, the risks are not leaving anywhere, increasing the estimates and taking way more time for QA engineers to adapt to paired testing.