GamerAI

A local agent that plays Scrap Mechanic from screen pixels at 20 Hz on one RTX 3070, plus the self-updating recorder app that collected its 45-hour training set.

2026 softwareAI recorder downloads ↗

GamerAI is an agent that plays Scrap Mechanic on my PC from the pixels on the screen, pressing real keys and moving the real mouse. I started building it on 24 July 2026, on one RTX 3070: a Windows recorder app that four testers ran and recorded with, a labelling and training pipeline over 45 hours of recorded play, and a fine-tuned Open-P2P policy that plays live at 20 steps a second and computes a step in 5.7 ms on an idle card. The policy walks, looks and clicks but does not finish tasks, and I measured why. It is the vision half of an idea from my 2023 language model, which could launch programs but could not see the screen.

Plumbing and a speed budget, 24 to 27 July 2026

For the policy I chose Open-P2P, a behaviour-cloning model for keyboard-and-mouse games that Elefant AI published in January 2026 (arXiv 2601.04575), trained on more than 8,300 hours of human play. Each step it takes a 192×192 frame and a one-line instruction, keeps the last 200 steps as context, and outputs up to four keys, two mouse buttons and a mouse movement.

On 24 July the plumbing passed against Scrap Mechanic at 2560×1440. Windows SendInput delivers scan-code keys and mouse-look with no kernel driver. DXcam grabs a frame in 4.7 ms and the downscale to 192×192 takes 4.2 ms, so capture uses 8.9 ms of each 50 ms step and leaves the policy 41.1 ms.

On 27 July I benchmarked the configuration Elefant AI calls 150M (183 million parameters, 10 transformer layers, 3.0 GB of VRAM). In eager PyTorch a step took 170 ms for about 3 GFLOP of arithmetic, because attention fell back to an unfused path. Compiled with torch.compile, through the community Triton build for Windows, it took under 20 ms.

Engine state from a Lua hook, 27 to 30 July

The slow half of the design is a text-only language model on the CPU (Qwen 3.5 9B through Ollama, 8 to 14 seconds per turn) that reads game state instead of pixels and writes the policy’s next instruction. The state comes from a Lua hook that an installer patches into the game’s own SurvivalPlayer.lua as three marked lines. The Lua sandbox has no file or socket access, so the hook writes JSON into the game log about five times a second (position, view direction, speed, health, held item, inventory changes), and Python tails the log. The log’s timestamps have one-second resolution, so each record carries the game’s 40 Hz tick counter.

For the first 21 recorded sessions the hook wrote nothing. On 29 July I found the cause: Scrap Mechanic 1.0 serves its game scripts from a cache and ignores edited files unless it is launched with -dev. Terrain scripts are exempt, and every check I had used to prove that edited scripts load had been a terrain script. With -dev, the first 60-second test logged 317 records with no errors and a median 215 ms from game to Python. -dev also disables multiplayer, so engine state exists only for solo play.

First fine-tune and first live play, 30 July to 26 August

I wrote a fine-tuning pipeline around the vendored inference code, and the first run trained on 30 July on about two hours of my recordings. By 3 August the loop was closed: the agent captures the game, runs the policy and injects its actions through SendInput. On 6 August I timed it during real play. With the game rendering on the same card it ran at 16 steps a second; after resetting the key-value cache every 200 steps, warming the kernel cache and killing orphaned compile workers, it held 19.0 steps a second with a 49 ms median step.

On 13 August the policy joined the planner, the telemetry reader and a gate that decides whether a line is worth saying, as blocks of one agent that run at their own rates and share one store of current state. On 26 August speech recognition (faster-whisper) and a local voice ran in the same process for the first time. Having the planner steer the policy live is work in progress.

A recorder four testers ran, 3 to 31 August

To collect play from more people than me, on 3 August I packaged the recorder as a Windows app (PySide6, PyInstaller); on 4 August it gained ten interface languages and shrank to a 145 MB zip. It starts recording when the player enters a world and stops when they leave. It captures at 60 fps, keeps every third frame at 192×192 plus a 720p archive on the same frame grid, logs keys, mouse buttons and mouse movement per tick, and can record narration from the microphone. Each session is packed into a bundle that records what the player agreed to share, ready to upload.

One tester sent the first outside recording on 4 August. On 5 August I sent the call for recorders over Discord to friends and people I had already played Scrap Mechanic with, explained the project and asked whether they wanted to help. By 10 August the two testers active at that point had both rebound their controls, so every session now saves the player’s own key bindings and training maps them back to one action layout. Since v1.0.0 on 12 August the app offers each new release from a public GitHub release channel and installs it with one click: a dropped download resumes where it stopped, and a failed swap into the install folder rolls back.

v1.1.1 on 17 August fixed a gap in the data itself. Desktop Duplication, the capture API under DXcam, returns frames without the mouse pointer, so a policy could never see its cursor in a menu. The recorder now draws the real cursor bitmap back into every frame at three times life size, so it survives the downscale. Scrap Mechanic shows an open hand over a slot and a closed fist while an item is dragged. In the first recording with shape tracking, one of the three cursor shapes had the left button held on 92.8% of its ticks, which identifies it as the fist from the input log alone.

The four testers installed each release as I pushed it, and I used their feedback to improve the app. By 24 August, when the last outside bundle arrived, they had sent about 8 hours of recordings, close to a fifth of the final corpus. v1.3.0 on 31 August added the game’s own audio.

Labels from narration, the hook and a vision model, August

The policy reads each step’s instruction as one 768-number text embedding, so every recorded second needs a free-form English label on a time span. The labels come from three sources: narration transcribed with Whisper (two thirds of it in Russian, translated to English), inventory events from the hook, which name the item but not the verb (“collect the wood”), and a vision-language labeller I scored against the game’s own event log. On 24 August a specific instruction covered 51% of the contributors’ labelled time and 44% of mine. By 31 August, 45.5% of all labelled time carried a specific instruction rather than “play survival”.

What the policy does with the screen, 27 August to 1 September

On 27 August I measured why the policy does not finish tasks, by destroying one input at a time and re-scoring the predictions of run15, my fine-tune of 26 August, against a recording of tree chopping. Blacking out the screen changed the mouse-x prediction on a third of frames but cost only 4.3 points of accuracy (43.0% to 38.7%); a real action history taken from a different clip cost 16.2. The screen is read and barely used. Across the training data, 89% of frames repeat the previous action, and the previous action is an input, so copying it is the cheapest way to lower the loss.

run16, trained on 31 August, added an auxiliary loss that predicts a 24×24 sketch of the screen 2 to 16 frames ahead. On 1 September I replayed 400 frames of a recording neither run had trained on, with eight seeds each. run16 held a key or button on 67.8% of frames where run15 held one on 6.7% (p < 0.0001), the first difference between two checkpoints large enough to stand clear of the seed noise. Almost all of that is one sustained mouse button: the held set changed on 1.8% of frames, where mine changes on 39.1%. The same test showed that running two checkpoints on shared random seeds does not reduce that noise here, because replayed footage leaves the sampler as the only source of randomness.

A faster step, faster training and a packed corpus, September

On 1 September I found that the hook had stopped writing on 30 July without any error, and reinstalled it. The 125 sessions since, 43 hours, carry no engine state, so only 5.2% of the corpus does. The recorder and the corpus report now warn when a session has none.

On 3 September I rebuilt the inference step. The vendored key-value cache grew by concatenation every step, so tensor shapes changed for the first 200 steps and the step could never be captured as a CUDA graph. I replaced it with a fixed ring buffer that overwrites the oldest step in place, and split each step’s two transformer passes into 4 and 8 tokens instead of 12 and 12. One step became one CUDA graph launch, and 19.1 ms became 8.1 ms. The fused attention kernel then took 3.3 ms of that, walking 38 key tiles in series for 4 to 8 queries; explicit matrix products with an fp32 softmax brought the step to 5.7 ms, 3.4 times faster than the original cache. Over 260 test steps with the same random draws, the ring produced the same 2,080 action tokens as the original cache, including after it wrapped.

On 4 September training moved to 8-bit AdamW. The step had been waiting on memory with 0 MiB free on the 8 GB card, not on compute, and went from 1.2 s to 0.5 to 0.6 s; once VRAM was also reclaimed before each validation pass, it held 0.32 s. run18 then trained a full epoch, 12,578 steps, in 1 h 19 min. It tested a fix for the copying shortcut, replacing 30% of the input action history with “nothing pressed” during training, and the mouse predictions got worse.

Raw 192×192 frames cost 7.4 GB per hour of play, seven times the 720p archive of the same footage. On 10 and 11 September I repacked the corpus losslessly as H.264 (libx264rgb): the first 144 sessions went from 359.5 GB to 115.4 GB (3.11×), and all 149 still stored raw finished with zero failures, each decoded and compared byte for byte twice before its raw copy was deleted.

On 23 September the play loop got a deadline pacer that holds 20.00 steps a second, where the old sleep-based loop held 19.80. After three duplicate copies were removed, the corpus went from 154 sessions to 151: 45 h 39 min, 3,287,538 frames. Open-P2P was trained on about 180 times as much play.