We Are Here for Quality: PropellerAds QA Team on Quality Assurance, Control, and Testing Practices

PropellerAds, a part of AdTech Holding, is one of the biggest advertising networks worldwide. It relies much on its smart technologies allowing advertisers access to global audiences and publishers to the most advanced traffic monetization methods. Thus, all products and services must work faultlessly — and it is the main concern of the Quality Assurance team. 

Generally, Quality Assurance is a set of practices that helps developers and the whole product team make sure a product or a feature works as it should. In very simplified terms, a QA engineer tests software and reports on errors and faults so that developers can timely fix them. However, this process is much more complex and challenging than laymen might suppose.

We spoke to Vasiliy Petukhov, the Head of QA, and Mikhail Sidelnikov, the Lead QA of the department Product Team who shared their insights on how to build efficient workflow, find a balance within a QA team, and feel certain about the product quality.

---

On the role QA plays for tech products

— Why is quality assurance so important for a big Adtech company like PropellerAds?

Mikhail: 

— PropellerAds is an enormous ad network, with more than 1 billion audience coverage and 25 million tracked conversions per month. This fact implies some very challenging points. First, such projects always have a massive RPS (requests per second). There are plenty of people using the Internet, seeing and interacting with ads — especially in the midst of some big events like the World Cup. 

Second, the response speed means really much: nobody wants to wait until we check all ads and distribute them before users. 

Finally, every mistake might cost too much: a bad deployment can quickly and significantly affect the company’s profit in a bad way.

Vasiliy: 

— PropellerAds is a good example of how the right QA practices can affect revenues. When we came to the company back in 2016, it turned out that it totally lacked any QA culture and strategy. There was no QA department; nobody discussed QA in the planning stage or used any test environments or convenient tools. 

Overall, there was no system in place, and all issues were described as ‘Something doesn’t work: we need to fix it’. It led to significant revenue losses. ‘We lost about $70,000 per week’ was a pretty common thing to hear. Obviously, we needed to make things different and prevent the company from losing money. That’s how the story began.

---

PropellerAds approach to QA in 2016: what was wrong?

On the QA definition

— How would you explain QA, and what does a QA team do?

Mikhail: 

— Generally, people understand QA as a process of testing: a QA engineer checks if some behavior or action meets particular expectations. It’s perfectly right, but it takes approximately 5% of the whole work that QA engineers really do.

Vasiliy:

— Our vision is that QA is based on three pillars: the process, the team, and the infrastructure. The process is about the reasons for testing, the right time to perform it, and the overall approach to this work. A team is about people who ensure the product quality, and the infrastructure includes all tools, test environments, and automated testing software — everything you can’t do without if you want to ensure good quality in the end.

---

Three pillars of QA

On the QA Process

—  What is the main goal of quality testing? Is there anything but making sure the product works fine?

Mikhail: 

— Yes, the first thing we need to do is to set particular goals. Of course, checking that product works okay is important, but it’s not the major QA focus. 

The main and basic aim of QA is to receive and share a transparent status of the product at any time.

As I have already mentioned, a QA engineer does not only test the product and decide if it does work as expected or not. In fact, they are responsible for the overall quality, so they must know and understand the current status of a product or a particular feature, and share this status anytime they are asked to. 

---

— How to arrange a process of testing that will stay convenient and transparent?

Vasiliy:

— It’s pretty simple. We have a particular set of goals that we must complete to have a working framework. This is how it looks at PropellerAds:

• Separating functional areas — all product features, microservices, and how they interact with each other. In other words, you break down the whole system into several pieces and test every piece separately in more detail.

---

How to separate functional areas

• Building auto and manual testing teams. It’s essential to have experienced and responsible people who can both write a test case for some particular unit, and build test automation that will work and deliver correct results.

• Implementing QA in the development process.  Quality Assurance processes are essential at all product stages, so QA employees are a must in any product team. 

QA at all development stages

Planning stage: understand what you will need to test
Requirements stage: describing QA tasks and developing the Definition of Done or Acceptance Criteria
Development stage: implementing Test Driven and Unit testing practices, and manual tests on development environments
Testing: the process of testing itself
Deployment and post-deployment: making sure a feature works as it is supposed to

• Building an efficient CI/CD process. Waterfall models are passing now as they don’t allow you to stay agile and develop the product really fast. The point is to complete a task, and test it quickly and properly.

CI/CD approach that allows making any number of releases daily

Code: you can quickly build a code and deploy it in a test environment
Set of tests: you have tests that work quickly and efficiently and run them after every change in a system
Result reports: you receive clear results and deliver them to the right places
Deployment: you have all tools to make the deployment process quick and easy
Monitoring: you have monitoring systems correctly set on the production environment

• Implementing an efficient set of tools. It includes automated testing frameworks, bug trackers, test management systems, repositories, and build system synchronization. All these tools allow you to boost transparency, control every step, and immediately see if something goes wrong.

---

— So, is it a ready-to-go checklist?

Mikhail:

— Yes, that’s about right. Most teams can follow the same strategy and mirror this approach. There is always a space for improvement for every point of this checklist: you can always speed up or better visualize some processes.

---

On QA Teams

— What is your approach toward a QA team?

Vasiliy:

— Think of some famous soccer team that won just every single match they played. All of such teams are based on the same ever-working approach: they have similar ever-winning rosters. Just like football clubs, we created a flawless team structure for the QA department.

---

A line-up for a QA team

Head Coach: The QA Head

• Organizes the process, communicates to every team member from a developer to CEO, implements top-level tools, shows how important the QA process is;
• Solves internal issues and conflicts, using a personal approach to every team member, works on relationships within a team;
• Sees the full image of the system, not just a particular team or project;
• Reports to senior management.

Star Players: Experienced QAs

• Decide on how to organize and automate the processes;
• Create an infrastructure for everyone: optimize CI/CD workflows, implement quick creation of test environments;
• Build the basis: basic units and frameworks that allow quicker and more efficient automation building;
• Provide team members with convenient tools that create transparency. Such tools may include reports systems, test results notifications, and test management systems;
• Help and educate less experienced team members.

Rookies: Junior QAs

---

Such a lineup always works out — no matter if you play some sports game or develop IT products. We at PropellerAds believe that vast experience makes a perfect match to the thirst for knowledge and brings unbelievable results.

---

— So now you know what the team will look like. But what do QA engineers do in a team and what skills do they need?

Vasiliy:

— A QA engineer is a kind of one-man army. They play different roles, and every QA engineer at PropellerAds is deeply involved in the product and all its stages, from planning to deployment.

Roles of a QA engineer

QA = Test Engineer — checks and shares a transparent status;
QA = Developer — analyzes the code, finds issues, implements unit tests;
QA = DevOps — creates a testing environment, deploys;
QA = Product Owner — helps to create the right tasks, including the most unobvious ones

What does a QA engineer do at every product stage

Planning: helps to analyze a task, determines the Definition of Done, elaborates the testing process
Development: prepares testing environment and data, plans and creates a test base
Testing: revises and upgrades tests, performs testing, shares testing results to developers
Deployment: performs testing in production, monitors results, runs end-2-end tests, checks Slack chats and makes sure there are no issue alerts

---

On the QA Infrastructure

— How do you create and maintain your infrastructure?

Mikhail:

— We developed our own set of tools and processes that create a convenient infrastructure. It includes tools for storing, executing, checking, monitoring all tests, and tracking statistics. These tools help to understand how to write and run tests, keep testing convenient and transparent, share results that are clear to everyone, and overall make the whole system handy.

Storage: Allure TestOps. It allows to quickly integrate autotests in the system and keeps releasing new features that help to ensure quick CI/CD.

Execution: Set of practices. Here PropellerAds has another convenient checklist on how to write a code, where to store it, and when/where to run it. This approach allows automating workflows and reduces time input for routine tasks:

• Language — Java, as it offers a vast selection of various solutions, frameworks, ready parallel tests, and more;
• Storage — a code is kept a repository with a product, which allows working within a single task;
• Tests — start upon a trigger from the pull request;
• Test environment — created right on a testing server, and is deleted after the testing is over — no need to keep unnecessary environments
• Results — in the Slack chats and Allure TestOps

---

— You suggest that transparency is the top priority of QA. What tools do you use to maintain it?

Mikhail: 

— We use ready solutions like Slack messenger to share testing results. Besides, we created our own tools for monitoring changes and statistics. 

Sharing results:

A particular notifications system for the Slack messenger that collects data from various systems. 

It sends a clear message containing the following information:

• Test results
• Allure report link
• Teamcity link: to check logs in case of issues
• Pull request link in Github
• Test logs links

The team can discuss all issues in the same Slack thread with managers and engineers which is very convenient.

Monitoring:

A tool that contains all information on deployment and changes in the system.

It has handy search filters by commands, elements, and time, allows an engineer to check what task the test relates to, its priority, and a link to a release on Github. There is also a special feature that collects data from various systems responsible for business metrics.

This feature highlights potential problems and thus allows us to quickly find and solve potential problems that may lead to losing profits.

Statistics and tracking:

A special tool that allows monitoring all data in a catch-all interface and tracking all essential QA stats in a single place.

PropellerAds creates a quickly growing product, thus it’s essential to constantly monitor all results: which tests were written for a particular module within a certain period, how many tests were run, and what was the latest result.

 The tracking tool aggregates data from Allure Testops and Github and allows the team to vividly see all information. For instance, the tool can show that there are 140 tests for a particular unit, 30 tests written within a quarter, and 25 pull requests made in a particular repository, with part of them crashing due to failed tests. This information is very important to understand what can be improved and what are the current weak points.

---

On the results

— Did all the work produce the results? What has changed since you came to PropellerAds and implemented the brand new approach to QA?

Vasiliy:

— Our main achievement is that we managed to make all the processes transparent, clear, and of high quality. We worked stepwise and built the whole system gradually — and it brought significant results. In short, we are now sure the product meets our standards, generates value for our customers, and brings us profits.

Mikhail:

— All that we did was make a big change: the company stopped suffering from large losses and can control all possible issues before they even arise. After all, it was all about small and comparatively easy steps — no rush and pressure.