Pytest is the king
of Python testing tools.
With more than 12,000 stars on GitHub,
a very active community, continuous improvement with new releases,
and a lot of forks and plugins to extend its functionality,
pytest is the most important reference
when we need to test code in Python.

If you look for Python testing tools or frameworks on the Internet,
you’ll find articles
like “Python Testing Frameworks”
by E. Sales,
“10 Best Python Testing Frameworks”
by GeeksForGeeks,
“Top 9 Python Testing Frameworks”
by M. Echout,
and many other blog posts with similar rankings
where pytest is never missing.

The comparison with other tools seems unfair
since pytest is a tool for unit and integration testing,
whereas other frameworks compete in a very specific domain.
Lettuce
and Behave,
for example,
introduce behavior-driven development,
but they’re not for all development teams;
Robot works for end-to-end tests and RPA,
but it’s not for unit or simple integration tests;
Testify,
TestProject,
and others are dead projects…
The unittest module
(a Python built-in module)
and nose2
could be the biggest competitors to pytest,
but they lack the support, community, and plugins pytest has.

At Fluid Attacks,
we use and recommend pytest.
Still,
we decided to wrap it before using it in our codebase.
In this post,
I want to share why you should try it
and how we achieved a new testing framework
for highly maintainable and readable tests.

Testing with pytest

A previous blog post,
“From Flaky to Bulletproof”
by D. Salazar,
guides our intention to improve the unit and integration tests.
There,
we recommend the use of a testing module
(a pytest wrapper)
due to a big deal:
standardization.

When your development team has a lot of members,
you need to define some rules to speak the same language.
You can add linters and formatters to standardize the code syntax,
but the way programmers write code is more tricky to standardize.

Some libraries are considered “opinionated”
because they enforce standardized usage with specific file structures
and their own methods and classes.
Indeed,
pytest has its own methods and files,
but it is so flexible that it seems “unopinionated.”

These are some of the most unwanted features of pytest
when it comes to its flexibility:

• Pytest has an official way to mock methods and classes,
  but you can mock using other libraries (e.g., unittest) without conflicts.

• Pytest fixtures actually do magic.
  You can modify all test behavior with fixtures
  without directly referencing those functions
  and messing up the expected test flows.

• You can put tests wherever,
  even beside the functional code.

• You can use real or mocked services
  because pytest execution is not sandboxed.

If you assemble a growing team
in which everyone has different experiences working with pytest,
you can accumulate tons of technical debt,
unreadable code, and exponentially increasing WTFs/minute.

Therefore,
we decided to implement our pytest wrapper,
which includes other testing libraries like moto, freezegun,
or coverage-py to write test code in only one way.

Let’s talk about our new guidelines and their benefits:

Pytest prohibition

Thanks to the importlinter library,
we forbid the use of pytest and unittest
in any place other than our testing framework.
We safeguard against misuse by removing pytest fixtures, unittest mocks,
and any unnecessary features that might be introduced later.
Thus,
the testing module can use pytest
and export the most important tools to be used.

Wrapped utilities

pytest.raises (to catch exceptions),
pytest.mark.parametrize (to handle multiple cases per test),
and freezegun.freeze_time (to use a fake time to run the test)
are the most common features that we use.
We found a way to wrap them into functions
that can be imported from our testing module,
making them easy to use and allowing us to document examples
of how to implement them.

Mocked services

Fluid Attacks’ platform uses AWS services.
We decided to mock them with moto
to achieve very simple and fast tests
thanks to in-memory simulation of services like DynamoDB and S3.
However,
moto requires some boilerplate code to run and ensure isolation among tests.
Developers could add complex logic inside tests to mock services
or forget the right way to clean up these mocks
because some of the copy/paste steps miss it.

For that reason,
we included a decorator in our testing module
to start the fake AWS environment.
We also encapsulated all the startup and cleanup
to ensure test isolation and simplify the tests.
Developers only need the documentation
to know how to preload data or files for testing
using a standard declarative approach.

Fakers for data objects

Leaving behind the discussion
about the differences between fakers, mocks, stubs, and spies,
we needed to create fake data.
We addressed this by implementing Fakers,
a collection of functions designed to return specific fake objects
based on the data types.

Fakers are simple to implement and can call other fakers
to populate nested fields.
Developers can modify any field when calling them for flexibility.
This approach significantly reduces boilerplate code,
enabling the creation of well-structured test objects
without the need to define every property manually.

Coverage per module

We took a deep dive into the coverage-py library
to achieve a modular solution.
This powerful tool generates detailed reports on test coverage
for the executed code,
serving as a critical resource for identifying gaps in our test suite.
By analyzing these reports,
we can check areas where additional tests are needed,
and the modular approach helps to focus developers on important tests first.

Also,
the new file structure speaks for itself.
Test files are next to normal files,
giving developers a simple way to check for missing tests and extend them.

Continuous discussion

If any case requires one of the pytest missing features,
any developer can start a discussion to validate if a new feature is necessary
or if the current tools can handle the case.

We are open to discussions and improvements with the whole team,
prioritizing testability and readability.
Any component must be easily testable,
and any test must be highly readable.
We embody a declarative (explicit decorators in the startup)
and descriptive (tests are divided by Arrange, Act, and Assert sections)
solution for testing.

Results

Our testing framework allows developers to test
if a user can create a new organization
(collection of groups or projects to be assessed) on our platform:

They can also test if a file was uploaded to Amazon S3:

It was an amazing change
because the amount of WTFs/minute that old tests could generate
was considerable,
even more so when tests had to be maintained
and some mocks could hide potential bugs:

Thus,
we get a very simple and readable code with our testing framework.
The developers greatly appreciated the new testing experience
and have frequently commented on the ease, speed, and confidence
with which they execute the new tests.
That good experience reduces the assertion-free tests problem progressively
and motivates developers to write important tests.
We even focus some weeks on migrating old tests to the new framework
in a hackathon style,
prioritizing our code quality to be faster afterward.

Note:
V. Khorikov in Unit Testing: Principles, Practices, and Patterns
called “assertion-free tests” those tests that don’t verify anything.
Even with asserts at the end,
a test could assert things that are not significant
(e.g., a function was called n times
instead of querying a new entity added directly from the database).

A continuous testing practice is the first layer of any security strategy.

In this blog post,
I shared our learnings around building a pytest wrapper for standardization
and its huge benefits.
Don’t forget that enforcing standardization,
reducing features to the essential ones,
mocking your external services only,
and being open to team discussions and feedback
can improve your testing culture.
The more testable and readable your code is,
the faster you can deliver new value
(and the more confident you’ll be in your production releases).

Special thanks

• D. Salazar,
  for supporting, discussing and implementing the core of this solution
  with me.

• D. Betancur and J. Restrepo,
  for attaching and prioritizing the testing framework development
  to the roadmap.

• The development team,
  for using the testing framework, giving meaningful feedback
  and contributing to its maintenance and growth.