Flutter: Golden testing

Photo by Lucas Benjamin on Unsplash

Introduction

Flutter offers a wide range of automatic tests, such as:

• unit testing, suitable for individual methods and/or classes
• widget testing, handy when checking the visual components of our app
• integration testing, useful in order to review end-to-end flows or features
• golden testing, for pixel-perfect tests

So golden tests are available by default in the Flutter framework, but their usage is not as frequent as the other types of testing. What are they exactly? When should we use them? Are they worth it? Let’s find out!

What are golden tests?

Golden tests are basically automatic UI regression tests. If we split this definition into different parts:

• they are automatic, so they can be scheduled and included on any testing pipeline
• they are focused on the UI, checking the Look&Feel rather than the logic of our app
• they belong to the regression test family, preventing us from making unintended visual changes when modifying a widget

Motivation

How many times have you messed up the alignment of a Column, shrinking all its contents by accident…? Do you remember playing around with some Opacity or Visibility widget that ended up hiding the contents of the app…?

Golden tests can detect these changes easily and provide useful feedback to solve them when required.

How do they work?

Golden tests are based on a series of screenshots of our app called “golden files“. These files are basically bitmaps that depict the desired visual appearance of the application, so they are considered the “one and only” source of truth when it comes down to any UI related matter.

When testing our app, Flutter compares the current appearance of the application with these templates and reports back any differences found as errors/failures.

Since Flutter uses composition and every application is basically a large tree composed of several nodes, each separate widget can have its own golden test based on a particular golden file.

Workflow

Adding golden tests to any Flutter app is quite simple! We only have to:

1. Implement the golden test
2. Create the required golden file
3. Run the golden test
4. If errors are reported back, perform a “Red-Green-Refactor” until all errors are solved

The following sections describe each one of these steps in detail.

1. Implementing a golden test

Golden tests are run in the environment provided by the default flutter_test library, so their entry point is the “testWidgets()” method. This method takes as parameters:

• a string describing the test
• a code snippet containing the test itself

void main() { 
   testWidgets('The 1st golden test...', (WidgetTester tester) async { 

      // 1)
      const GOLDEN_FILE_NAME = "my_app.png";

      // 2)
      await tester.pumpWidget(const MyApp());

      // 4)
      await expectLater( 
         find.byType(MyApp), // comparison is asynchronous    
         // 3)
         matchesGoldenFile(GOLDEN_FILE_NAME)); 
   });
}

As we can see in the previous example, the test:

• 1) specifies the name of the golden file the test is going to be compared against
• 2) pumps the widget under test
• 3) uses a specific matcher (“matchesGoldenFile()“) that takes the golden file name as parameter
• 4) it finally compares the former golden file with the current widget and returns a boolean value. This operation is performed asynchronously, so we use the “expectLater()” assertion

Piece of cake!

2. Creating the golden files

The golden files templates are created by running:

flutter test --update-goldens

This command generates or updates (when already created) all the golden files specified in our test suite.

Generated files are included by default in the “test” directory of our app.

The generated .png file seems a sketchy wireframe of the app, but it contains all the data required in order to accept or reject the test.

3. Running tests

As usual, these tests are launched with the command:

flutter test

4. Error reporting

While testing the app, errors are reported back on the command line.

Additionally, each error is assigned a deviation percentage, representing the “amount of difference” between the current layout of the app and its corresponding golden file.

Furthermore, a “test/failures” directory is created. This folder includes the following screenshots:

• <widget_name>_masterImage.png: the ideal appearance for the widget under test, according to its golden file
• <widget_name>_testImage.png: the current appearance for the given widget

Golden tests under the hood

Golden tests basically do some image comparison in order to determine if the test passes or not. The Flutter framework “captures” the current UI and then compares it with the associated golden file at byte level. For each byte, both the pixel represented and its encoding metadata are checked.

The former comparison is performed using an instance of the “GoldenFileComparator” class by default.

Additionally, we can use our custom comparator used when running our tests. In order to do so, we have to:

1. create a comparator subclass extending the comparator super class

class CustomGoldenComparator extends GoldenFileComparator {
  @override
  Future<bool> compare(Uint8List imageBytes, Uri golden) async {
    await Future.delayed(const Duration(seconds: 5));

    //TODO: add proper implementation, performing byte comparison
    return true;
  }

  @override
  Future<void> update(Uri golden, Uint8List imageBytes) {
    //TODO: add proper implementation, performing byte comparison
    throw UnimplementedError();
  }
}

1. specify our comparator in the code, setting the goldenFileComparator property in the test file:

We can also specify a comparator using the command line:

flutter test goldenFileComparator=<path_to_custom_comparator>

Other features

As mentioned before, golden tests are another type of test automation, so they can be included on a CI/CD pipeline.
Many integration tools like Codemagic offer support for golden testing.

Flutter packages for golden tests

Although golden test are included in the Flutter framework by default, the Flutter community have created some libraries that extend the default behaviour and offer additional functionalities.

The golden_toolkit package, for instance, adds extra features that may come in handy when working with accessibility or widget variations.

Code repository

You can find a golden test sample in the following link:

https://github.com/begomez/FlutterGoldens

References

https://flutter.dev/docs/cookbook/testing/widget/introduction

https://www.youtube.com/watch?v=_G6GuxJF44Q