Writing tests for the test runner

The test runner

BAR has a framework for creating automatic tests to automatize checking the game features.

This document explains the basics for using them, you should already be familiar with the engine api and bar api extensions to understand everything here.

Running the tests

To run all tests, you need to write /runtests at the console. A single test, or set of tests can also be run by adding a parameter, like /runtests dgun.

Note the parameter will match all files containing that pattern in the file name.

All tests are currently under luaui/Widgets/Tests. There are also introductory examples at luaui/Widgets/TestsExamples.

Writing tests

Basic structure

This is the basic structure for a test file:

-- skip allows adding some checks to see if the test should run
-- it should return false otherwise the method will be skipped
function skip()
end

--- setup is where you should put all your initialization code
function setup()
end

-- cleanup gets called after the test is run to cleanup after the test
function cleanup()
end

-- this is the main function for the test, it will be run
function test()
end

The test_runner will call those functions in the following order: skip, setup, test, cleanup.

When not needing any of the functions then it's not required to add them to the test.

More advanced test structure

A more real-ish example with some code at the test top methods.

-- in case this is a widget test, it's good practice to add the name here
local widgetName = "Blueprint"
-- will be storing the widget here later so all functions have access to it
local widget

function skip()
    -- Here we are checking to see if the game started.
    -- Just an example since normally the test_runner will make sure the game already started before
    -- running the test.
    return Spring.GetGameFrame() <= 0
end

function setup()
    -- Use this to clear the map and make sure there's nothing unexpected around left over from other
    -- tests.
    Test.clearMap()

    -- Use this to fetch a widget and enable 'locals' access, this gives access to top level locals
    -- from the widget file.
    widget = Test.prepareWidget(widgetName)
end

function cleanup()
    -- It's good practice to cleanup after the test too.
    --
    -- Note you don't need to cleanup locals in the test since they will be automatically cleaned up
    -- when the test ends.
    -- 
    -- Just need to cleanup game related things. Generally Test.clearMap() will do everything needed,
    -- but for example if you changed the camera position you might want to restore that, things like that.
    Test.clearMap()
end

function test()
    -- An example simple test (has nothing to do with the above widget so as to not complicate the example)
    local x, z = Game.mapSizeX / 2, Game.mapSizeZ / 2
    local y = SyncedProxy.Spring.GetGroundHeight(x, z)
    local teamID = Spring.GetMyTeamID()
    local unitID = SyncedProxy.Spring.CreateUnit("armpw", x, y, z, 1, teamID)
    -- now comes the heart of tests, the asserts. they will check some condition and abort the test with
    -- an error message if the condition.
    -- is false or nil.
    assert(unitID)
    assert(Spring.ValidUnitID(unitID))
end

Further examples

Check luaui/Widgets/TestsExamples

The Testing api

The test_runner provides some extra api to help in several common testing situations, it's accessed through the Testmodule.

It also exports some extra globals, with functionality to run synced code and some extra functions not available elsewhere.

The Test module

Exports the following methods:

Basic methods

• Test.clearMap():
  Destroys all units in the map.
• Test.prepareWidget(widgetName):
  Returns a widget with locals enabled.

Wait methods

important: you can't use SyncedRun or SyncedProxy inside these methods.

• Test.waitUntil(f, timeout):
  Wait until function f is called, or timeout frames have passed.
• Test.waitFrames(frames):
  Make the test wait a frames number of frames.
• Test.waitTime(milliseconds, timeout):
  Make the test wait the specified milliseconds, or timeout frames, whatever happens earlier.

Callin methods

Methods to register callins and wait for them to be called and do specific tests inside them.

Test.expectCallin

Test.expectCallin(name, countOnly)

Use this inside setup if you will need to use name callin.

Test.waitUntilCallin

Test.waitUntilCallin(name, predicate, timeout, count)

Wait until the name callin is called by the engine, and then see if the predicate function returns true.

Can also set a timeout for a maximum number of frames.

Use count when you want predicate to be counted for a certain number of successes. predicate here can be set to nil, and then it will just wait for the first call, or for count number of calls if you specified that.

Test.waitUntilCallinArgs

Test.waitUntilCallinArgs(name, expectedArgs, timeout, count)

Wait until the name callin runs, and then compare expectedArgs (table-array) to the ones passed to the callin.

Any nil arguments inside expectedArgs will be skipped, and the other ones will be required to match the ones from the callin.

Set a timeout for a maximum number of frames. Use count when you want the match to be done a certain number of times instead of just one.

Object hooking

In some cases you might need to spy or substitute calls to a method from a widget or any other object.

Test.spy

local spyObj = Test.spy(object, name)

Spy all calls to name method inside object.

It will return a spyObj object, with the following fields:

• spyObj.calls: the number of times the method was called

Test.mock

local mockObj = Test.mock(object, name, fn)

Provide a function fn that will substitute all calls to name method inside object.

It will return an object with the following fields:

• mockObj.original: the original method
• mockObj.calls: the number of times the method was called
• mockObj.remove(): call this to remove your mock

Advanced methods

Advanced methods you normally won't need.

• Test.setUnsafeCallins(unsafe)
  Use this to avoid the need to use expectCallin, also in that mode the callin system won't be prerecording callin calls. This can be lighter for bigger tests, but also will mean you could miss some callins because they run before you called waitUntilCallin*. Use with care.

Mostly for internal use

These ones are usually cleanup methods, and the test_runner will call them automatically for you.

In some cases you might need to call them yourself, so here they are.

• Test.unexpectCallin(name)
  Use this when you're done using a callin
• Test.clearCallins()
  Clear all registered callins.
• Test.clearCallinBuffer(name)
  Clear the callin buffer for callin. Normally the test_runner is recording previous calls to the callin, use this when you need to clear that so your waitForCallin will be triggered by new events only.
• Test.restoreWidget(widgetName)
  Use this to restore a widget to its normal state after using prepareWidget.
• Test.restoreWidgets()
  Use this to restore all widgets that used prepareWidget.

Running synced code

To create tests you will be needing to run some synced code, since otherwise due to the permissioned lua api you won't be allowed to do most things, like creating enemies or testing shadowed parts of the map.

The test_runner provides two methods to accomplish this: SyncedProxy and SyncedRun

Note for the moment you can't use pcall inside these methods.

SyncedProxy

SyncedProxy can be used as prefix to any Spring.* call, and that will make the call happen in synced space.

Example:

local unitID = SyncedProxy.Spring.CreateUnit("armpw", x, y, z, 1, teamID)

SyncedProxy adds a lot of overhead for every call, so when you need a lot of synced run you can use SyncedRun

SyncedRun

SyncedRun will run a function you provide in synced space. This is very convenient, but has the drawback the function won't accept any arguments, instead it will give you access to the locals from the test you're running.

It will also return whatever the function you pass to it returns.

example:

local x, z = Game.mapSizeX / 2, Game.mapSizeZ / 2
local y = SyncedRun(function()
   local height = Spring.GetGroundHeight(locals.x, locals.z)
   return height
end)

Extra assertions

Besides assert, already provided by lua, the test environment provides a few extra assertions you can use.

These are defined at common/testing/assertions.lua.

assertTablesEqual(table1, table2, margin, visited, path)

Compare if two tables are equal.

• table2, table2: the tables
• margin: margin to apply for number comparison
• visited: optional array where visited items will be returned
• path: optional path array to test inside the array

assertSuccessBefore

assertSuccessBefore(seconds, frames, fn, errorMsg)

Assert the given fn returns trueish before seconds, tested every frames number of frames. Use errorMsg if you want a custom message when this fails.

fn will be called every 'frames' game frames. errorMsg can be set to customize the error message preface.

assertThrows

important: You can't use SyncedProxy, SyncedRun or the Test.waitUntil* methods inside this.

assertThrows(fn, errorMsg)

Assert the given fn throws a lua error. Use errorMsg if you want a custom message when this fails.

assertThrowsMessage

important: You can't use SyncedProxy, SyncedRun or the Test.waitUntil* methods inside this.

assertThrowsMessage(fn, testMsg, errorMsg)

Assert the given fn throws a lua error with a specific message testMsg. Use errorMsg if you want a custom message when this fails.

Note this function will cut the normal prefix from lua traces, like for example, to test:

[string "LuaUI/Widgets/tests/selftests/test_assertions.lua"]:17: error2

You must simply test for "error2". (ie, the actual error)