How MessageMedia uses feature toggles

This week we have a guest post from Binh Phan, a Technical Lead at MesssageMedia Vietnam.

Binh is currently the technical lead of the Conversations API team at MessageMedia in Vietnam. He spends his time coding and helping the team run smoothly. As a techie, he enjoys sharing knowledge and listening to experienced developers.

At MessageMedia we have implemented a Continuous Delivery process to roll out new features frequently. To reduce the risk of deploying code changes in our production environment, we release new features by applying the following approaches:

  • Canary release: A new feature is only available for a small group of users. Based on user feedback, the feature can be improved and then rolled out to a bigger user group.
  • A/B testing: Variants of a new feature are available to different user groups. The user feedback to each variant is used to select the best solution.

In order to facilitate these release approaches, we decided to use a technique called ‘feature toggles’. This article highlights our learnings and best practices when using different toggle types for product development.

What are feature toggles?

Feature toggles are used to configure the behaviour of an application. Instead of deploying a modified version of an application, the behaviour can be easily changed while the application is running. A simple example of a toggle is a boolean switch that enables or disables a particular feature.

We distinguish between the following types of feature toggles:

Release toggles

Used to disable and enable product features without the need for code changes. While a feature is still in development, a release toggle can be used to deactivate the incomplete and partially tested code. This approach avoids negative impacts on the production environment and allows efficient management of available features.

Experiment toggles

Used for managing A/B tests.

Ops toggles

Used for monitoring and operating the application. For example, a specific performance problem can be traced by temporary switching on an Ops toggle. After completing the investigation, the toggle will be deactivated.

Permission toggles

Used to enable and disable features for specific user groups. For example, a toggle can prevent a free user from accessing an advanced feature, which was developed for paid users.

How we use feature toggles

During the development phase of FlickPay, a simple mobile bill payment solution of MessageMedia, we used the following toggles to improve our Continuous Delivery process.

Release toggles

We designed an enhanced user interface to manage billing information of FlickPay. The development team estimated eight weeks for implementing, testing and delivering the expected modifications. Following our approach of Continuous Delivery, we avoid long-living code branches and big bang deployments. Therefore, the development team introduced a release toggle to quickly activate and deactivate the user interface changes. In the testing environments, the changes were activated so that all stakeholders could verify the results. In the production environment, all user interface changes were deactivated so the continuous deployment of code changes didn’t negatively impact the live product. Once the testing results met our expectations, the product manager made the changes available to all users.

Experiment toggles

Customers of FlickPay receive SMS messages to initiate the payment process. Understanding how the users interact with these messages is fundamental to realize a product, which is straightforward and easy to use. The development team created an experiment toggle so that the product manager could easily activate different message content for specific user groups. The collected usage data helped to select the best content for each message type. As a result, the engagement of existing customers improved and the sign-up numbers of new FlickPay users increased significantly.

Permission toggles

Our software delivery life cycle for FlickPay includes manual and automated quality tests. An essential set of test cases covers sending and receiving SMS messages. To prevent that test messages are accidentally sent to a real telephone number, the trigger for the SMS delivery can be deactivated with a permission toggle. In our testing environments, this toggle is deactivated so that the SMS delivery is restricted to a whitelist of destination numbers.

Togglz

We are using the open source library togglz.org to manage the feature toggles of all services. It provides a user interface for accessing the available toggles so that non-tech team members can easily activate, deactivate and configure the toggles. Togglz has activation strategies for configuring the conditions of enabling and disabling a feature, see screenshot below. Our developers benefit from togglz because they can easily integrate it into Java Spring/Spring Boot applications.

Challenges and solutions

Introducing toggles for FlickPay has helped the development team to deliver better software faster. But after using toggles for several months, we have been facing the following challenges.

Too many toggles

As time progressed during the development of FlickPay, many toggles have been created. The development team could hardly remember the purpose of each toggle and if they were still needed. Therefore, we have defined the following team rules for managing toggles:
• Keep the number of toggles manageable
• Create documentation for the right usage of toggles
• Proactively remove toggles, which are no longer needed
• When creating a new release toggle, always add a backlog task for removing this toggle in the future

Increased code complexity

The high number of toggles we created for FlickPay, resulted in increased code complexity. Our developers and QA Engineers had to spend additional effort in implementing and executing quality tests.

For instance, a simple release toggle can be implemented with the following if/else statement:

if <feature A is enabled> then
[run feature A]
else
[Throw an invalid feature error]
end

The additional code branch for the feature flag has to be verified with unit tests and manual tests. By removing the toggle, the additional if/else statement is no longer needed so that the code is easier to read and to maintain. To reduce the negative impacts of the code quality, we avoid conditionals as much as possible:

  • Inside business logic, we use Aspect Oriented Programming instead of if/else statements. The development teams are using annotations in Java to define the decision points.
  • We apply the Polymorphism characteristic of Object Oriented Programming combined with the Factory pattern to minimise the use of if/else statements.

Conclusion

By introducing feature toggles, we have improved our delivery process in several ways. Everyone involved in our software delivery process benefits from the use of feature flags:

  1. Developers can deliver code changes faster so that they don’t have to maintain long-living code branches
  2. Quality Assurance Engineers can verify new features in various environments without adverse impacts to our customers
  3. Product Managers can verify new feature ideas and then collect feedback from selected user groups

However, feature toggles should be used carefully to keep the code maintainable.


Continue reading...

Managing API documentation over multiple teams
Bec Martin
September 12, 2019
Your first Django project
Natalie Byrgiotis
July 08, 2019