Flutter: test coverage

Photo by Fabe collage on Unsplash

Introduction

Test coverage is a code metric that gives us some useful insight about a given project. Depending on the coverage level, our confidence in a source set may increase or decrease.

However, our code will never be bug-free, even when having a high coverage level. But coverage is an important point to consider when designing a testing strategy, for instance.

Project configuration

Dependencies

Flutter framework is shipped with some useful tools when it comes down to testing.

In order to enable testing in our project, we just need the following library:

dev_dependencies:
  flutter_test:
    sdk: flutter

Running tests

Once we have configured the project, tests are launched by executing:

flutter test

The previous command executes all the test files contained in the “/test” directory.

Once it’s done, we get a summary of successful/erroneous tests:

The test command has several options, but in this case we will focus on the coverage property:

flutter test --coverage

This option basically keeps track of the lines of code executed (aka “covered”) when running our tests. By the way, a complete command reference for the command can be found here.

When executing our tests with the coverage option, the resulting output is a “lcov.info” file. Problem here is that this format stores encoded data, so it’s not very user friendly…

lcov format

Introduction

Lcov is a graphical tool that collects raw data about tests (generated in the previous step) and transforms it into a set of structured HTML pages containing coverage information. It also contains some handy command line interface.

Installation

To install LCOV (on Mac), open a new command line and run:

brew install lcov

That would be all! More details available at the lcov homepage.

Execution

After installing lcov, we can execute a new set of commands related to test coverage, like the generate html command:

genhtml <source_file.info> -o <target_file.html>

Where do we get the “*.info” source file…? Well, we got it out-of-the-box when invoking the flutter test command with the coverage property. Under the hood, when calling flutter test with the coverage option, we’re just invoking the geninfo command from the lcov library.

Skipping components when covering

Lcov also allows us to “ignore” certain files or complete directories when keeping track of the coverage in our project. When generating the raw data, any file or directory ignored will not be taken into account.

This is useful, for instance, if we store the autogenerated classes of our project in a certain folder. We can also ignore, for instance, a simple directory containing only enumerations and constants (no logic to test there).

In order to customise the directories we want to cover, invoke:

lcov --remove <input_file.info> <files_or_directories_to_ignore> -o <output_file.info>

When ignoring some elements, we can use the * wildcard to build regular expressions. For instance, we can ignore the autogenerated files with the following pattern:

*.g.dart

Scripting for automation

We can use some automation tool like make in order to group all the previous commands into a useful task:

flutter test --coverage

lcov --remove coverage/lcov.info "***/constants*/**" "**/*.g.dart" -o coverage/lcov_cleaned.info

genhtml coverage/lcov_cleaned.info -o coverage/html

open coverage/html/index.html

Plugins

Additionally, there are several plugins that allow us to integrate the test coverage option directly into our IDE. For instance:

https://marketplace.visualstudio.com/items?itemName=Flutterando.flutter-coverage

The plugin adds a new panel into our workspace can be used to review the coverage data, either grouped (by package) or individually (by file).

Some other plugins go even further and actually embed the coverage data inside the text editor, so every line of code is highlighted on different colors, depending on whether it is actually covered or not.

Code sample

As usual, check the following repo for the complete source code:

https://github.com/begomez/Flutter-Arch-Template

References

https://pub.dev/packages/test_coverage