Cover image by Tanya Denysenko

How We Handle Quality Assurance (QA) in Stormotion

Nikita Ivashchenko

Nikita Ivashchenko

Rockstar Copywriter @ Stormotion

Ostap Shtypuk

Ostap Shtypuk

QA Engineer @ Stormotion

*“The bitterness of poor quality remains long after the sweetness of meeting the schedule has been forgotten”, — Karl Wiegers, an American software engineer*

We believe that this quote hits home for almost every competent IT specialist, and Stormotion is not an exception at all. Obviously, it’s impossible to achieve high quality without testing the product and that’s where Quality Assurance comes to help.

Quality Assurance is a method to determine whether the software is matching the requirements of the stakeholders at the current stage, as well as to “polish” the product to the point of being as close to defect-free as possible. Although exhaustive testing is impossible, its conduction is hugely beneficial in terms of cost-efficiency, product quality, and customer satisfaction.

The clients are usually also interested in proper testing of their product. In general, QA Engineer finds and confirms around 80-90 bugs for a project of medium complexity. In bigger projects it’s rational to expect more than a hundred bugs appearing. Imagine releasing a product of such quality, keeping in mind that there are also smaller bugs that aren’t making it to documentation. We don’t think that such a project is going to fulfill the customer's business needs, so Quality Assurance is without a doubt vital for success.

QA Engineers are so-called hunters in a project’s jungle full of various bug species.

Unfortunately, a completely bug-free project is an impossible task for anyone (image by Ana Beverin)

In general, the bugs found throughout the testing are not the result of developers being incompetent or them making wrong decisions. Usually, they appear in situations that can’t be predicted and are only seen after implementation. For example, it may turn out that some requirements are absolutely incompatible in a single project, or the level of required customization supposes configurational changes that may lead to unforeseen results. Also, we can’t forget about the bugs caused by the appearing technical debts and the team’s ways to handle them.

If you are interested in Stormotion’s way of solving technical debt issues and some general tips from our experts, please make sure to check the article down below.

Read also:

How to Deal With Technical Debt?

In this article, we’ll try to get acquainted with Quality Assurance in general, its processes, challenges, participants, and also open the curtains of Stormotion’s inner QA flows along with sharing our best practices from our experience. Let’s get to the point!

 
 

🙋 Who Takes Part in QA at Stormotion and What are They Responsible For?

The main specialist in Quality Assurance in our team is without a doubt our QA engineer. However, there are people who usually also contribute to the process, leaving no place for bugs to escape 😼

So let's acquaint ourselves with everyone responsible for the Quality Assurance stage at Stormotion!

Developers. Our developers responsible for the project are the ones who start the “testing cycle”. Their task is to provide our QA engineer with a crash-free product version after conducting Smoke testing.

Smoke Testing is a software testing stage usually conducted by the developers to approve the product version’s readiness for the QA. Typically it’s done to check for the installation and database connectivity errors by running small tests.

Project Manager. In our team, the QA cycle is a so-called safety net. In case our developers are unable to conduct Smoke Testing, our PM is the one to provide the crash-free build — code pieces compiled into a program which is executable yet remains unreleased — to the QA engineer. Also after our QA specialist fills in the documentation, the PM always needs to verify it.

Project Managers in QA take the role of verifiers, who are responsible for both verifying QA’s documentation and received builds.

Although only QA specialists are usually called “testers” they’re not the only ones responsible for the project’s testings (image by Ana Beverin)

**Designers. Even though their role isn’t directly associated with the QA cycle, the design they create needs to be maximally convenient for the user. **To achieve the best UX, they should conduct usability testing in the process of creation, before delivering the app to the QA.

QA Engineer. The person whose role is to make sure that the delivered software build is of the highest quality. In brief, their responsibilities include:

  • Verifying the filling of the Product Backlog according to its rules and canons.
  • Making sure the UX/UI parts are convenient and ergonomic without any unfinished actions and features.
  • Ensuring every version is delivered on time along with creating a report for every Testing Round for the Tech Team to see the bigger picture.
  • After the build release, conduct Sanity Testing to proceed to the next QA stage.

Sanity Testing is a way to test a build’s sanity before taking the next step into more advanced testing. For example, it makes no sense to test complex features unless you are sure that every basic one is working correctly.

 
 

🔗 Steps of Quality Assurance Processes at Stormotion

A common misconception is to think that Quality Assurance (or it’s also called “Software Testing”) is a process described as a single activity. Vice versa, QA is a set of activities and methods that aim to complete the process of validating the product’s stability and verifying its usability. As Software Testing is a complex process, it’s usually referred to as Software Testing Life Cycle (STLC) for specification.

In this section, we’re about to present Stormotion’s internal processes for STLC, taking our Norsk Guardian project as an example.

Norsk is an IoT companion app that our team managed to develop in about 30 weeks. Based in the USA, Norsk Lithium develops and sells batteries for outdoor activities including open-water fishing, ice fishing, and camping. In this project, the development itself along with the testing stage took about 18 weeks.

The Software Testing Life Cycle at Stormotion consists of 5 phases instead of the common 6, as in our team the Environment Setup Stage is not included as a separate one. Here are the phases of STLC our QA Engineer always goes through:

  1. Requirement Analysis
  2. Planning
  3. Case Development
  4. Execution
  5. Cycle Closure.

_Although Quality Assurance is a parallel process with the project development itself, it doesn’t usually start right away. The best approach is to involve a QA engineer as early as possible to gather more useful information on the project’s planning stages. _

Let’s take a look at each of them separately!

 
 

Requirement Analysis

Requirement Analysis — also known as Requirement Phase Testing — is a stage when the project requirements are reviewed through the prism of testing. This phase helps our team to understand testable requirements as well as to determine automation feasibility for testing.

Automation Feasibility is a process of determining test cases that may be automated based on the given functional requirements and availability.

As Norsk Guardian is a project more inclined to manual testing as some main features such as Bluetooth can’t be automated. Thus, after discussing with the team, we decided to save time by sticking up to manual testing instead of an automated one.

Common activities for the Requirement Analysis stage are:

  • Identification of testing types to be performed. The QA engineer should discover the requirements and decide which types of tests will be performed.
  • Verification of the backlog structure and user stories. Software requirements are usually expressed as user stories. The QA engineer should verify the backlog structure and avoid such issues as the UI design not matching the context of the user story. Also, the QA engineer should verify that every user story is uniquely identified, complete, traceable, testable, and prioritized.

One of the Norsk user stories stated that users should be offered to set up a notification reaching a custom charge level. When verifying the backlog and user stories, the question was where to locate the required notification block in the UI.

Shot from the Norsk app

Shot from the Norsk Guardian app

A "Set Custom Charging Level Notification" button should be found after proceeding to the "More Options" section in client’s "Batteries Group". Its location between the “Delete Boat” and the “Rename Boat” features is necessary in order to perform regression tests in a row. A QA engineer doesn't need to move to another section and spend extra time on these actions.

  • Verification of UX/UI design. At this stage, the QA engineer verifies the conformity of UX/UI, performing Usability testing to avoid illogical patterns. Also, this is when accessibility testing can be performed.
  • Identification of test environment details where testing is supposed to be carried out. The Quality Assurance engineer should prepare templates for the main documents which will be used on the project (Test plan, Traceability matrix, Checklist, Test cases docs).
  • **Automation feasibility analysis. **The necessity of automation testing implementation should be directly discussed with the team. On some projects, automation testing shouldn't be required.

 
 

Test Planning

Test Planning is a stage for QA specialists to determine the strategy for a test plan, testing schedule, cost estimates, test environment, and limitations. All the risks should be also determined as well as The Test Plan should be finished by the end of this phase.

As in any other spheres, planning stages play crucial roles determining the great project flow.

Test plannings define the main methodologies and estimates that a QA Engineer will use throughout the tests (image by Outline2Design)

Common activities for the Test Planning stage are:

  • Preparation of test plan/strategy document for various types of testing. The QA engineer should analyze the requirements and then decide which type of testing will be used to deliver a high-quality project.
  • Testing tool selection. This stage supposes a QA Engineer to determine all the tools they are going to use in the process.

Among the testing tools for Norsk Guardian our QA Engineer used Slack, Google Sheets, Active Collab, TestFlight, Figma, Android Studio, and others.

  • Test effort estimation. At this stage, QA specialists with the PM themselves create an estimate containing the task, the ones responsible for it, and the required number of hours.

On the Norsk Guardian project, we identified the main functionality, which we would cover through tests. Sometimes, in the case of a low functionality project, we would cover by tests 70-80% of all functions, but on large projects, the test cases cover only 20-30%.

 
 

Test Case Development

Test Case Development stage is about creating new and modifying already existing test cases and scripts after finishing the Test Plan.

A test case example

A test case example

Test Case is a testing artifact that contains a set of actions and/or conditions required to test the functionality of the software part

Common Activities for Test Case Development:

  • **Test case creation. **The structure of test cases at Stormotion is described in the Master Test Plan. When the "test case development" step is completed, we start working on a traceability matrix.
  • **Reviewing test cases. **The Test Cases should be reviewed by another QA engineer to verify that they include everything from the functionality.
  • Traceability matrix creation. Requirement Traceability Matrix (RTM) — is a document used to map and track the test cases representing user requirements.
Requirements matrix

At Stormotion the RTM matrix is used to verify that all our requirements are covered by test cases. This document we present along with the Master Test Plan to the stakeholders.

 
 

Test Execution

Test Execution supposes exactly the process of test script implementation and bug reporting after the test plan is finished. The bugs reported to a development team are getting fixed for the further iteration of test execution.

Common activities for Test Cycle Closure are:

  • Executing tests after the test plan. While filling out the Test Plan document our QA should decide when and what types of testing to use on the current project. Therefore, during the execution, our specialists should stick to the plan they've chosen before, unless there are required changes from the customer. In QA plannings customer’s changes also suppose changes in testing documentation.

On the Norsk Guardian project, our QA decided to include localization testing as the project will focus not only on the English-speaking market.

  • Retesting the fixed build. As soon as the updated build reaches our QA Engineer, they are performing a retest on the fixed build.
  • Reporting the status of each iteration. The QA engineer should create a report after every iteration of Testing. After that, they perform a feature or regression testing as soon as possible to give our developers time to fix some issues before the Demo.

Regression Testing is a testing method used to determine whether the new code modifications have affected the results of previous tests. Usually, it makes QA engineers retest new builds using previous successfully implemented iterations.

For convenience, the document for QA logs was created to track the time of verifying the version and understanding whether the build is stable. The report should include the build version, a number of issues, fixed bugs, and the list of “Open” bugs.

  • Manage the project's status in the QA logs doc. At Stormotion we have a document for QA logs which allows us to track the status of all projects. This document was created to help our development team to understand whether the version is stable as well as to structure the development phases.

 
 

Test Cycle Closure

Test cycle closure is a phase where obviously the testing cycle and its execution are coming to an end. With filling the test completion reports, this is when the QA team gathers and analyzes information regarding the processes of this project. After the discussion, it becomes clear what aspects weren’t as good as they could be and what to do in the next projects to enhance efficiency.

The QA cycle closure doesn't mean the beginning of the celebration yet, as the results should be properly analyzed and presented

Test Cycle Closure is the final part in a QA cycle consisted mostly of evaluations reports and analysis of the conducted testings (image by KiraSvitlana)

Common activities for Test Cycle Closure are:

  • Evaluation of cycle completion. At this stage, we need to analyze the results and check whether the product meets our minimal expectations noted in a special document. So, after analyzing the Test Cycle, the Test Report is created containing:
    • The number of found bugs
    • The number of bugs with status Opened
    • The statistic of found bugs divided by severity
    • The percentage of bugs found/fixed
  • **Preparing Test closure report and its analysis. **After creating the report QA Engineers prepare it to present to the development team to fix the bugs along with analyzing the outcome on their side.

 
 

🔒 What are the challenges in Quality Assurance?

Generally, customers that aren’t deeply aware of the inner flows of QA, tend to underestimate its importance and the amount of work threat the QA Engineer is facing. Such misconceptions usually lead to making it harder for the specialist to give their best results.

We'd like to share the main challenges a QA Engineer may face during Quality Assurance based on Stormotion’s experience:

Challenge

Possible Solution

Frequent changes in customer’s project requirements make a Team rewrite documentation and choose a different testing approach.

We recommend you to spend more time thoroughly understanding and prioritizing your requirements before the development starts. It may not deflect all the needs in their changing in the future, but will definitely make them less crucial for the Team.

Clients require the highest product quality without willing to spend more hours on testing it.

The quality of your product has its own price. Thus, be ready to allocate extra resources if you have high technical requirements.

Cases, when Quality Assurance estimates are not high enough to stick to without losing in a single aspect. Thus, QA specialists have to test in a fuss to keep up, increasing the risks of missing out on something.

To avoid such challenges it’s better to structure your budget with a complete understanding of the QA level you require as a stakeholder of the project. As well as to be ready for having to decide the need of raising the estimates during the development.

Projects, where stakeholders have no clue about the importance and basics of the Quality Assurance stage.

As you are already acquainting yourself through reading this article, this challenge is less likely to hit your Team. However, keep in mind that the more you understand this topic, the easier it’ll be to establish great communication with your Tech Partner and achieve your goals.

Vacations or delays from either a client or Team members. As Quality Assurance requires developers and QA to work in synergy, every gear in this mechanism is extremely important for productivity. And vice versa, in case the Team needs any approvals while the customer is unavailable — the development and testing may have to be stopped.

To avoid this challenge’s appearance we recommend you to be more precise in sharing your working schedule with a Team. The earlier you notify about your vacations or days off the easier it’ll be to adjust the testing cycles for the times you are present.

The main challenge for a QA Engineer is to reach the closest point to the ideal product version after Quality Assurance. As the testing principle states that an absolutely bug-free project is utopian, QA’s task is to get it to the highest possible level.

 
 

⌛ How much time does QA usually take?

As every single project we build is a unique experience, there is no precise answer to this question. In general, the QA stage starts almost from the very beginning of development. And till the product release Quality Assurance becomes its inseparable part.

There are always risks that are not present in every single case that, however, should be considered. The time required for the QA stage depends on the unique methodology our QA engineer chooses before the conduction. This methodology includes an exclusive Testing Plan based on the project type, its risks, and the customer’s behavior as a stakeholder. If a client is constantly making changes in his requirements for the project, they are also making its life cycle significantly longer.

Based on our experience at Stormotion, the QA cycle usually takes up to 20-30% of the whole project’s lifecycle, making it quite a noticeable part of the whole development in the bigger picture.

 
 

📈 How can you make Quality Assurance more effective as a stakeholder?

We mentioned in previous sections that stakeholder behavior requires consideration while creating Testing Plans for the product. But what should a customer consider to make the QA effective and time-saving?

The more you as a stakeholder are present throughout the development the bigger are the chances for a product to fulfill all of your business needs.

You as a stakeholder can also assist your Tech Team in raising your product’s quality (image by Anna Magura)

Here are some tips from our team on how to achieve a maximum of QA productivity:

  • Be actively involved in the development. Spend a little more time launching the product yourself, and don’t only rely on demos. Our team provides project guides and tests every single version to make it convenient for our customers to even test it themselves.
  • Be available for the team as much as possible. In case you have additional questions or updated requirements it’s always better to provide them immediately. And vice versa, whenever the Team requires extra clarification, you are able to make the processes more efficient by being available more often.
  • Determine your precise requirements for the future project and try to avoid changing them too often.
  • Don’t be afraid to ask more questions and discuss everything regarding bugs and QA right away.
  • Thoroughly acquaint yourself with the project documentation. Our team always provides a Test Plan and a list of potential risks for the client to be aware of the processes and the testing order.
  • In case the project supposes connecting several different devices and we are unable to have them all in our office, we recommend launching new tested versions on them as well.

 
 

❓ What to do if a customer has their own QA Engineer on the Team?

There are cases when customers already have the QA engineer in their team and are willing to take control of the process. Generally, we see three potential scenarios of how it can be done:

 
 

QA is strictly conducted by our specialists

This is the most common scenario from our experience. Usually, a client doesn’t have any objection to letting our QA engineer do their job. We highly recommend entrusting the Quality Assurance to the team responsible for the development, as it is more convenient in such ways:

  • Greater communication levels. When there’s a strong link between QA, developers, designers, and a PM, it drastically enhances the productivity of the Quality Assurance stage. Communication between the team members is built up by the constant back-to-back project development and can’t even be compared with the other scenarios in terms of efficiency.
  • Matching time zones. There will be no trouble in additional handling time zone differences among the team members.
  • Unique approach. Every Tech Team has its own boundaries, visions, and approaches to a project. The most ineffective thing to do in such an environment is to trust an outsider expecting him to enhance productivity. It often becomes quite the opposite, taking more time to adapt and understand the approach and priorities which the Tech Team had been working on from the beginning. Thus, making the QA stage much longer and more expensive.
  • No language barriers. There’s also no need to seek convenient ways for communication among the team members in terms of a preferred language.

 
 

QA is conducted by the client’s specialists

This scenario is certainly less common and the risks it brings compared with the first case clearly show why. Letting a customer’s side to conduct a Quality Assurance stage usually implies such risks:

  • Chances of exceeding the estimates are significantly higher.
  • Required additional time for a specialist to adapt to the development team and its workflow.
  • Possible time zone differences, which creates inconvenience both in communication and development.
  • The team must speak the language of the specialist provided by the customer, playing a crucial part in making communication and adaptation even harder.

 
 

Compromised QA conduction

It may sound like nonsense, however, unlike the previous one, this type of QA stage even has a way to be efficient. This case may be used in larger projects, where the scope of work is noticeably bigger. With the compromised QA conduction on both sides of the development, it becomes possible to find bugs and problems way faster.

However, the risks are not leaving anywhere, increasing the estimates and taking way more time for QA engineers to adapt to paired testing.

You are the one to determine the most fitting scenario, however, choose wisely in order to achieve the maximum efficiency

Although, the combined QA conduction is also a real case, we still recommend to entrust its processes to your Tech Team (image by Jourdan Rahschulte)

Obviously, the less risky and more productive QA is always when it is inseparable from the Tech Team. Therefore, we recommend not to reinvent the wheel, wasting time trying to make it efficient, and completely trust the Team you are hiring in every aspect.

 
 

💡 Takeaways

In conclusion, we’d like to emphasize one more time that the Quality Assurance stage, despite being assumed as an easy process, is quite complex and significant for the software quality. At Stormotion almost every team member takes part in Quality Assurance in one way or another. Developers, designers, project managers and of course QA engineers are the ones responsible for creating a defect-free product.

It’s also important to understand the inner flows of the Software Testing Life Cycle to take a deeper look at how the process of polishing is taking place. At Stormotion Quality Assurance supposes the conduction of such phases:

  1. Requirement Analysis
  2. Planning
  3. Case Development
  4. Execution
  5. Cycle closure

It’s also possible for you as a customer to discuss the involvement of your own QA specialist, however, after the comparison of such scenarios we highly recommend letting the Team take care of it to avoid extra risks.

If you have any questions or need help with creating quality software for your IoT solution let us know. We will be happy to help you meet your business needs with an outstanding product!

Contact us

Read also

Iot Development Costs Stormotion

How Much Does It Cost to Develop IoT Software?

The Stormotion Team: What Makes Us Special?

How to Localize your Mobile or Web App

How can we help you?

Our clients say

Stormotion client David Lesser, CEO from [object Object]

They were a delight to work with. And they delivered the product we wanted. Stormotion fostered an enjoyable work atmosphere and focused on delivering a bug-free solution.

David Lesser, CEO

Numina