Introducing Flextesa: Robust Testing Tools for Tezos and its Applications

Links from this guide were updated on July 1, 2020

Testing plays an important and often overlooked role in developing software. Great applications need to account for the 'what ifs' and the 'could that happen' situations which are rare but should be handled. Within the past year, we've had the privilege of building three open source applications for the Tezos community - Tezos Wallet and Tezos Baking for the Ledger Nano S (and now Nano X), and Kiln, our user interface for baking, voting, and monitoring Tezos infrastructure.

Frankly, testing this software can be a time consuming challenge. For instance:

• How can we be sure Tezos Baking will prevent you from double baking or double endorsing?
• How can we know Kiln users can vote during each period of the amendment process?
• How can we be sure Kiln will bake through a protocol transition? What happens if the protocol does not get voted in? Will Kiln keep running the right baker and endorser?
• How can we recreate these scenarios for every release, especially when some of them require the network to be in a specific voting period or state?

Test networks are a valuable tool for answering these questions, but they often fall short of creating the scenarios necessary to release software regularly with confidence.

Introducing Flextesa

For the past several months we have been contributing Flexible Test Sandboxes (Flextesa) to the core Tezos repo to assist us with testing, and we hope all Tezos developers find it useful.

Photo Credit: Martin Vorel

Flextesa provides local, custom testing by creating configurable and scriptable sandboxes to meet specific testing needs. You can use it to spin up a network with one node or several nodes. You can include several bakers, or have blocks manually baked as necessary. Protocol constants can be adjusted so networks move as quickly (or slowly) as necessary. Accounts can be instantiated in the genesis block, transactions simulated, and funds sent to hardware wallets.

There are two main components to Flextesa:

1. The tezos-network-sandbox library (found at src/lib_network_sandbox): This is an OCaml library used to write test scenarios that deploy, manipulate, and inspect small self-contained Tezos networks while providing the ability to make those scenarios interactive through custom command-line interfaces.
2. One binary tezos-sandbox (at src/bin_flextesa): This uses the library above to provide a few testing scenarios within the Tezos repository. Some are partially interactive, others can run fully automated (including in the CI).

Using Flextesa

Documentation can be found here! Here are a few instances of Flextesa sandboxes:

• tezos-sandbox mini-net provides a network which runs bakers, endorsers, and accusers with a few customizable protocol parameters. Also, see the daemons-upgrade subcommand which is a similar sandbox but also makes the network go through a whole round of voting and a protocol transition.
• tezos-sandbox accusations provides three specific scenarios which make double baking and double endorsement accusations happen. Unlike the mini-net, this kind of sandbox advances the chain step-by-step (instead of running bakers) and makes use of more interesting network topologies to make branches happen deterministically.

Within either style of sandbox, you can configure protocol constants or scenario specific parameters like account balances or the number of nodes in the network.

Some specific examples of our scripts which take advantage of Flextesa:

• We wrote a wrapper around the Flextesa's Accusation to test Kiln's double baking notifications.
• Flextesa's Protocol Change is used to test Kiln's ability to bake through protocol transitions and voting during each amendment period with your ledger device through Kiln's UI.
• Tezos Baking Core Functionality Tests are a key part of our Tezos Baking regression tests.
• Tezos Wallet Voting and Batched Operation Tests checks some of the hardest features of Tezos Wallet to test.

Either with existing scripts or through the flexibility to create new ones, we're sure that many others in the Tezos community can find Flextesa useful. In particular, this can be beneficial for:

Protocol Development: A protocol amendment, and the transition from the current protocol to the proposed should be fully tested before it is proposed, as that must be the final version of the amendment for consideration. Flextesa can allow protocol developers to run custom sandboxes with their changes and test the transition to their protocol.

HSM Signer Development: Similar to how we have tested Tezos Wallet with Flextesa, signer developers can use Flextesa to confirm their high water mark works in all situations and quickly regression test after new development.

Smart Contract Development: Smart contracts can be deployed within Flextesa networks, and operations to these contracts can also be scripted to run automatically. The data that results from test runs can be saved for future test runs or scrapped every time in favor of a new run of the scripted test.

Application Development: Wallet providers and more can confirm how their data will display under common usage, or test against unlikely but relevant scenarios. For instance, what happens if a transaction is included in a block which is not included in the main chain, but then a branch reorganization results in the block being a part of the main chain? Flextesa can create this type of scenario reproducibly on demand!

Next Steps

We'd love to see others in the community contribute to Flextesa. Some opportunities include:

• Additional modules for testing smart contracts and high-level API servers which interact with nodes (we've already started this).
• Making interactivity even more flexible. For instance, it would be great if users had more ways of pausing the scenarios at given breakpoints to investigate the sandbox.
• Making it possible to run and manage sandboxes on 2 or more hosts instead of just locally on a single machine.
• Including backwards compatibility scenarios, where nodes are running different versions of Tezos software.
• Tighter integration with Tezos' event logging framework.
• Being able to save a snapshot of a sandbox and restart a sandbox from one of those snapshots.
• Allowing sandboxes to join other networks (such as alphanet or zeronet) instead of just their local ChainID.
• Making it easier for someone to deploy their own public test network with its own faucet. This could be an invaluable tool for protocol developers looking for robust community testing and buy-in for prospective protocol amendments.

Want to contribute? Please let us know so we don't duplicate efforts :)

Have questions? Please ask on Tezos Stack Exchange! We do our best to answer promptly.

Want to say hi? Send us an email (tezos@obsidian.systems) or tweet (@obsidian_llc)!

After receiving requests from the community, we have also set up a donation address for those who wish to show their appreciation fiscally - tz1gddTh1i6qpviichg4e2GiSoNQer4FyAiM