Which is the best approach for testing Flutter Apps

Question

I'm working for a Flutter App which relies on an API. We are thinking about a testing strategy and we would like to know which should be the best approach.

According to their documentation ( https://flutter.dev/docs/testing ) they have 3 levels of tests:

• Unit tests
• Widget tests
• Integration tests (Pump widgets new approach)
• Integration tests (Flutter driver old approach)

As we have limited resources, we would like to know what we should pickup first. Since until now, very few effort was put on testing.

Our situation is the following:

• Unit tests (50% coverage)
• Widget tests (0% coverage)
• Integration tests (Pump widgets new approach - 0% Coverage)
• Integration tests (Flutter driver old approach - Only a few test scenarios covered, the main flows)
• API Tests: 0% coverage on unit tests and functional tests

And we are not using any testing automation framework such as WebdriverIO + Appium.

We would like to know how much effort we should put in each of the Flutter test categories, and regarding Flutter integration tests, would it make sense to just have Integration tests with the new approach (Pumping every widget) or we would also need Integration tests (Flutter driver old way)?. Relying only on the integration testing using pump widget approach doesn't make us feel very confident.

Some options we are considering are:

• Strong API coverage (unit test and functional test) + Strong coverage on Flutter unit tests + Few Integration tests using flutter driver approach
• Testing pyramid approach : Lots of unit tests + Less amount integration tests using pump widget new approach ,API tests and Widget tests + Less amount of E2E tests (maybe using Integration tests using flutter driver approach or an external automation framework) and manual tests
• Just unit test + Widget test + Integration tests the new approach of pumping widgets, trying to achieve 100% coverage in each of the three.

We also think that maintaining integration tests the new way (pumping widgets) is somehow very time consuming as you need to have good understanding of the views and the internals of the App. Which might be challenging for a QA Automation guy who hasn't got too much experience with Flutter development.

Which of the Flutter automated testing categories I should cover first, unit, widget or integration testing? Should I use an external automated framework such as WebdriverIO + Appium instead?

Answer 1

First, at this moment, I would suggest to think about testing in the application perspective, not on Flutter, React-native or Native perspective, well, test pyramid and test concepts are not really linked to any development tool/framework, at the end of the day the app must do what it's supposed to do gracefully.

Now, on the strategy topic, depends on a lot of variables, I will push just some to this answer, otherwise I will write an article here.

There is some stuff to think about, even before writing the strategy:

• When we will test?
  • Local build and testing.
  • Remote build and testing (CI/CD stuff).
  • Pre merge testing (CI/CD stuff).
  • Pre production testing (CI/CD stuff).
  • Production monitoring tests (CI/CD stuff).
• Do we have enough resources?
  • At least one person dedicated person for testing and it's tasks.
  • VMs/computers hosted by your company or cloud providers to run the tests in a CI/CD pipeline.

On my previous experiences with testing, when you are starting (low amount of coverage), end-to-end testing are the ones that did show more value, why?

• It's mostly about the user perspective.
  • It will answer things like "Can the user even login on our app and perform a core task?" If you cannot answer this before a release, well you are in a fragile situation.
• Covers how the application screens and feature behave together.
• Covers how the application integrate with backend services.
  • Well, if it has issues with the API, it will most likely to be visible on the UI.
• Covers if data is being persisted in a way that make sense to the user
  • It might be "wrong" on the database, but for who is using it still makes sense.
• You don't need 500 hundred tests to have a nice coverage, but still this kind of test is costly to maintain.

The problem with the base (fast and less costly tests) of the pyramid when you have "nothing" is, you can have 50000 unit tests, but still not answer if the core works, why? For you to answer it you need to be exposed to a real, or near real world, unit doesn't provide it for you. You will be really limited to answer things like: "well in case the input is invalid it will show a fancy message. But can the user login?"

The base is still important and the test pyramid is still a really good thing to use for guidance, but my thoughts for you folks right now, as you are starting, try to get meaningful end-to-end cases, and make sure that they are working, that the core of the application, at every release, is there, working as expected, it's really good to release with confidence.

At some point the amount of end-to-end will increase, and you will start seeing the cost of maintaining it, so then you could start moving things down one step bellow in the pyramid, checks that were made on the e2e, now can be on the integration level, on the widget level, and etc.

Testing is a iterative and incremental work also, it will be changing as the team matures, trying to go strait to the near perfect world with it, will cause a lot of problematic releases, my overall point is, at first, try to have tests that gives meaningful answers.

Another note is: Starting in the top of the pyramid that is not supposed to be linked to any development framework (Flutter, react-native and etc) will also give you time to get up to speed into Flutter, while you are still contributing to e2e coverage, using things like Appium (SDETS/QA must have some familiarity with it) for example, could be a parallel work.