Test types

Narya engine supports three types of tests: invariant tests, property tests, and single-run tests.

Test contract

All test functions should be defined inside a test contract.

Test contract requirements

The contract could exist within any file under the project root.
Test contracts must inherit PTest from @narya-ai/contracts.
The contract could optionally have setUp() function defined within it. It will be called once at the beginning of all types of tests. If test functions share the same setUp() code, they could be defined in the same Solidity file. Those tests will be checked independently.

Invariant Test

Invariant test is to test something that is always true, regardless of the smart contracts states.

How engine run it

Deploy your test contract.
Call setUp() function (if defined) to deploy your target contracts and environments.
Call 0 or more arbitrary functions using an agent account.
Call the invariant function to check the state. Without commiting the transaction to the network
Go to step 3, repeat until the engine is confident about the coverage, or hit the time limit.

This function will be called multiple times during the test by our engine. Avoid having observable side-effects inside of it.

Test function requirements

A typical invariant test looks like this:

function invariantTotalSupplyShouldNeverChange() public view {
    assert(token.totalSupply() == totalSupply);
}

The function name needs to start with invariant
The function needs to accept no parameter.
The function needs to be publicly callable, which means either public or external. Other modifiers are not required here.
The function needs to have at least one assertion. See Assertions.

Example assertions

Assert that the ownership never changes, even when a non-owner account calls arbitrary functions, in arbitrary orders.
Assert that your protocol balance never drops drammatically, even when arbitrary accounts call arbitrary functions, in arbitrary orders.
Assert that when a user deposit, they could always withdraw at least a certain amount, even when other accounts call arbitrary functions, in arbitrary orders.

Property Test

Property test is to test some properties of the contract always meet certain conditions.

How engine run it

Deploy your test contract.
Call setUp() function (if defined) to deploy your target contracts and environments.
Determine a good input that is not tested yet.
Call the test function to check the state, with the new input.
Go to step 3, repeat until the engine is confident about the coverage, or hit the time limit.

Test function requirements

A typical property test looks like this:

// Test if the `transferFrom` function behaves as expected, regardless what the `amount` is
function testTransferFrom(uint256 amount) public {
    uint256 aliceBalance = token.balanceOf(alice);
    uint256 agentBalance = token.balanceOf(agent);
    uint256 allowance = token.allowance(alice, agent);

    try token.transferFrom(alice, agent, amount) {
        assert(amount <= aliceBalance);
        assert(amount <= allowance);
        assert(token.balanceOf(alice) == aliceBalance - amount);
        assert(token.balanceOf(agent) == agentBalance + amount);
        assert(token.allowance(alice, agent) == allowance - amount);
    } catch {
        assert(token.balanceOf(alice) == aliceBalance);
        assert(token.balanceOf(agent) == agentBalance);
        assert(token.allowance(alice, agent) == allowance);
    }
}

The function name needs to start with test
The function needs to accept at least one parameter. The engine supports multiple arguments with all common argument data types, including complex ones like address, struct and array.
The function needs to be publicly callable, which means either public or external. Other modifiers are not required here.
The function needs to have at least one assertion. See Assertions.

Example assertions

When you write a test, and we wish to cover all the possible values for a certain parameter or property or variable, it's a good sign that you should use property test to cover all possibilities.

Assert that transfer works as expected regardless who is the sender, receiver, or what's the amount.

Single-run tests

Single-run tests are the most common Foundry tests. The engine will simply call the function one time, just like what Foundry and other test framework would do.

How engine run it

Deploy your test contract.
Call setUp() function (if defined) to deploy your target contracts and environments.
Call the test function to check.

Test function requirements

A typical single-run test looks like this:

function testTransferFrom() public {
    payable(address(safe)).transfer(1 ether);
    uint256 preBalance = address(this).balance;
    safe.withdraw();
    uint256 postBalance = address(this).balance;
    assertEq(preBalance + 1 ether, postBalance);
}

The function name needs to start with test
The function needs to accept no parameter.
The function needs to be publicly callable, which means either public or external. Other modifiers are not required here.
The function needs to have at least one assertion. See Assertions.

Assertions

The engine would treat the test failed, if the function is reverted in the end. This means all following ways would be recognized:

function invariant() {
    // 1. require:
	require(condition, "Invariant violated");
    // 2. if + revert
	if (!condition) revert("Invariant violated");
    // 3. assert
	assert(condition);
}

We recommand to use require() as it's simple and you could provide an error message to it.

Why prefer invariant/property tests over single-run tests?

When we are writing unit test and end-to-end (e2e) test, we have to come up with very specific test cases. These test cases are a subset of the total possible ones. The ones that we missed are usually where the bugs hide.

This is where the property test and invariant test can help. You can think of these 2 types of test are upgrade for the traditional unit test and e2e test.

unit test => property test
e2e test => invariant test

For property test, we randomize the arguments those are passed in. For invariant test, we randomize both the function call sequence and the arguments for each of the functions.

The word “randomize” is a modest way to describe it. But we are neither testing in a brute-force fashion, nor a continuous uniform distribution random-try fashion. What actually happens in engine is that, we use many metrics or features to guide the engine to search for a way to break assertions in the test functions.

The benefits comparing to the first generation of tests is not only the new way covers more test cases, it also makes writing test easier, as you don’t need to design and implement complex test cases. That why you would find the our tests looks much simpler in code. It is not because of they are simple sample code, in real life they should all look simple.

More examples

Check out our Github repo NaryaAI/narya-examples for more example tests.

Narya Docs