Active Vision System

The Active Vision system is implemented in the ActiveVisionEnv Environment, a base class for the games.

The environment emits two images, a small central, high resolution 'foveal' image, and a low resolution, noisy, full screen 'peripheral vision' image. The system is able to direct the 'fovea' (or gaze) to a grid position on the screen. The size of the grid is specified in the config file. The agent should not be able to solve the DM2S game by using the peripheral vision alone - it is not possible to make out the exact shapes, although their location is evident.

An example output from each is shown with a Tensorboard screenshot below (including the full screen image).

It is not enabled for M2S, but is for DM2S and M2L games. It can be enabled via the config files.

M2L is short for the game called MoveToLight (env name = m2l_v0). It is available to show the Active Vision system in operation.

Example commands to play the game manually and with an agent are shown below (using Docker - see Deployment page). The paths are specific to our machines, so you'll need to modify those when running.

Human gameplay of the active vision game

python keyboard_agent.py m2l-v0 configs/m2l_av_env.json

Run move_to_light (ml2) game with Active Vision

sudo deployment/run-docker.sh ~/agief-remote-run/WM_Hackathon/ ~/agief-remote-run/cerenaut-pt-core/ true python train_agent.py m2l-v0 configs/m2l_av_env.json configs/agent_env_av.json configs/agent_av.json

Pre-train (on m2s)

The pre-trained models are provided in the repo. However, if you want to change the hyperparameters and pre-train you are welcome. It involves generating input data, then training, as below.

Generate data

sudo deployment/run-docker.sh ~/agief-remote-run/WM_Hackathon/ ~/agief-remote-run/cerenaut-pt-core/ true python generate.py m2s-v0 configs/m2s_av_env.json 2000 ./data/gen_m2s_av

Pre-train fovea

sudo deployment/run-docker.sh ~/agief-remote-run/WM_Hackathon/ ~/agief-remote-run/cerenaut-pt-core/ true python pretrain_visual_cortex.py --config ./configs/pretrain_fovea.json --env m2s-v0 --env-config ./configs/m2s_av_env.json --env-data-dir=./data/gen_m2s_av --env-obs-key=fovea --model-file=./data/pretrain_m2s_av/fovea.pt --epochs 7

Pre-train peripheral

sudo deployment/run-docker.sh ~/agief-remote-run/WM_Hackathon/ ~/agief-remote-run/cerenaut-pt-core/ true python pretrain_visual_cortex.py --config ./configs/pretrain_peripheral.json --env m2s-v0 --env-config ./configs/m2s_av_env.json --env-data-dir=./data/gen_m2s_av --env-obs-key=peripheral --model-file=./data/pretrain_m2s_av/peripheral.pt --epochs 7