Testing Bramble

Tags:

• setup
• development

Categories:

• Development

The Bramble project comes with a suite of testing tools to help make the development process easy.

The tests are broken down into three categories: unit tests, integration tests, and fuzz testing. Integration tests are tests which compile and execute Bramble source code and validate the output. The Fuzz tester is a specially built tool which randomly generates Bramble source code for the Bramble compiler.

Unit Tests

The unit tests for Bramble are all included in the Rust source code, and can be identified by the source code file name containing the word test. Unit test modules are kept in the same directory as the module they are testing, rather than in a fully separate test module.

To run the Bramble unit tests, use the Cargo test command:

cargo test

If you want to run the tests against the release build:

cargo test --release

Integration Tests

All integration tests are run via a shell script; this makes it easy to run all the integration tests. The shell script will build a release version of Bramble and use that binary for all integration tests. The release build is used to make execution of the tests as fast as possible.

The integration tests are all stored in the ./tests with the source code for each test stored in ./tests/src and the scripts to run the tests in ./tests/. There are about 100 integration tests under the ./tests/src/ directory which the integration test will use for validation.

To run the integration tests, go to ./tests/ and then run ./test.sh.

There are three types of integration tests:

1. Positive integration tests: these test that features build and run correctly.
2. Failure tests: these test compiler errors.
3. Import tests: these test the compiler importing one Bramble project into another Bramble project.

Adding an Integration Test

All that is necessary to create a new integration test is to add the following files to the ./tests/src directory.

Integration tests consist of 2 to 3 parts:

1. The test file or project itself. This is one or more Bramble source code files.
2. The expected output file. This what the test source code should output if it runs correctly. This file has the same name as the test code but with .out as the extension.
3. Optionally, input data for tests that read from the console. This file has the test name with .in as the extension.

The test.sh script will automatically detect and execute your file. If you inlude an input file then the script will automatically pipe each line from the .in file to your test project.

Fuzz Testing

Testing all the possible syntactically valid Bramble expressions would be incredibly difficult if done manually. To avoid that work, a special Fuzz testing tool was written which randomly generates both syntactically valid and invalid Bramble code and tests that the compiler correctly handles it. Valid code should parse successfully and invalid code should generate a compiler error.

To run the fuzz tester:

cd ./tests
./test-syntax.sh

Currently, this only tests syntax and not semantics. The code it randomly generates will have undefined variables, functions, etc. As such, this fuzz tester only tests that Bramble can parse the code. The next step for the fuzz testing tool is to build a fuzz tester that generates semantically valid code which can fully compile and be executed.