Language model from scratch

A 72M-parameter dialogue transformer I trained from random weights on a dataset I assembled. It wrote Windows commands and Python expressions into its own answers for the app to run; below 0.8 confidence the app answered from a web search instead.

2023–2025 softwareAI

A desktop voice assistant built around a decoder-only transformer that I trained from random weights, on dialogue data I assembled, between September 2023 and August 2025. Most runs used a 72-million-parameter model. It answered in a fixed question-and-answer format and could put actions inside its answers: a Windows command after a <console> tag, or a Python expression in braces, which the app executed. When the model’s average confidence in an answer was below 0.8, the app discarded the answer and replied from a web search instead. The training app I wrote logged 45 runs between February 2024 and August 2025.

From a GPT-2 fine-tune to my own weights

It started in June 2023 as a fine-tune of pretrained GPT-2 (the small model, then the medium one), driven from a tkinter app and fed with wiki dumps, programming guides and books. A small crawler I wrote that month collected part of that text.

In September I moved away from pretrained weights. From 7 September, EGM (the “Easy Generative Model”) was a character-level LSTM written from tutorials, with a PyQt window for setting the network’s sizes and running training. By the 19th a variant of it ran GPT-2 medium for comparison. On 21 September I switched to a decoder-only transformer and trained its first checkpoint that afternoon. The model code, the training loop and the sampler are adapted from Vincent Bons’ public tutorial on building a text generation model; I wrote the dialogue data pipeline, the training app and its run log, the confidence score, the tool layer and the voice app.

Answers that contained commands

On 23 October 2023 I started SpeechHelperBot, a PyQt voice window with Google’s web speech recognition in and the Windows system voice out. By 29 October the transformer answered in it, its training text contained shell commands, and the window ran them.

Every training example is one question and one answer. The training code cuts each pair into one left-padded sample per answer token, so the model only ever saw text in that format. Two pairs from the main training set, translated into English:

<user> Launch the calculator
<text> Launching the calculator. <console> calc <endoftext>

<user> Exact time
<text> It is {datetime.datetime.now().strftime('%H:%M')}. <endoftext>

Anything after <console> runs as a shell command; for ipconfig the app parses the output and reads the IPv4 address aloud. By 2024 the app also evaluated anything in braces as Python, with datetime, time, random and sys in scope. There was no allowlist: the app ran any command the model wrote, which I only ever did on my own PC. Besides ipconfig, the commands open Chrome, Notepad, Explorer, Edge, the calculator, Task Manager, Word, Excel, PowerPoint and Photoshop, print the Python version, list Wi-Fi networks, or open Gmail’s compose page and YouTube searches. The expressions give the time, the date and the weekday, pick random numbers and do small arithmetic.

The tokeniser

The tokeniser began at single characters. On 16 November 2023 I tried OpenAI’s cl100k_base through tiktoken and byte-level BPE from Hugging Face tokenizers side by side, and kept BPE. By 29 February 2024 I had trained several BPE vocabularies on my own data; the one every run from March 2024 on used has 10,012 tokens. The special tokens <user>, <text>, <console> and <endoftext> are in it, and so are whole tool strings such as start chrome and datetime.datetime.now().strftime, each a single token.

A crawler that writes its own questions

The web part of the dialogue data came from UWEBS, a crawler I wrote between 21 November and 3 December 2023 and revised on 23 May 2024. It starts from one search query and takes the first 40 results. On each page, every h1 to h4 heading of two or more words becomes a question, and the paragraphs under it become the answer. Words that keep recurring across pages go into a queue and come back as new searches of the form “What is X?”, so the crawl grows its own topic list.

Offline voice and a confidence gate

On 5 February 2024 the voice app moved to offline speech recognition with Vosk and to Silero’s pretrained voices for speech output. The model itself was trained on one language. By 10 February English was added on top through translation: one MarianMT model turns the English question into that language and a second turns the model’s answer back into English.

By May the voice could be set to female or male, with four voices in the model’s language and ten in English. In that language past-tense verbs and some adjectives agree with the speaker’s gender, and the model answered in the masculine forms, so with a female voice the app rewrote them before speaking.

Generation samples at temperature 0.1 and returns the mean probability of the up to 150 tokens it produced, and the chat window shows that number beside the answer. In February 2024 a low score only turned the answer red. By 25 May it was a gate: below 0.8 the window discards the answer, runs a Google search for the question, takes the first result page with usable text and replies with a three-sentence extractive summary (sumy’s Edmundson summariser, with Luhn as a fallback), marked as a search answer with no score.

The same window saved each of my replies into a dataset file, paired with the bot’s previous answer as its prompt, so real conversations came back as training pairs with the roles reversed.

The training data

The main training set, in its last version of 10 July 2024, holds 7,645 question-and-answer pairs and 590,477 tokens. At least 2,900 of the pairs come from the crawler, about 1,150 are arithmetic questions and word problems, and 80 are those reversed real conversations. The rest are the tool examples, small talk and general-knowledge pairs, some of them written in my format by a chat model from a prompt I gave it. A small PyQt tool merged and de-duplicated the sources into one file.

The first version, on 25 November 2023, held 10 pairs. By 30 January 2024 it held 1,873, with 160 command examples for 29 different commands; by 31 March 6,258, and by 26 May 7,405, with 297 examples for 74 commands.

Forty-five logged runs

From 22 February 2024 the training app logged every run to JSON: the hyperparameters and, per epoch, 10 and later 20 readings of training loss, validation loss and accuracy. The window plots the same four curves live while a run trains.

The first 15 runs, from 22 to 26 February 2024, compared sizes from 4 layers at 512 wide (23 million parameters) to 12 layers at 768 wide (100 million). The model then settled at 8 layers, 768 wide (72 million parameters), the shape of 24 of the 45 runs. Six runs used 10 layers (86 million), two in October 2024 were 2-layer test models, and the last one, on 30 August 2025, used 16 layers at 512 wide (61 million). The logged models have a 32-token context (16 in one October 2024 run). The median question-and-answer pair is 56 tokens long, so for 69% of the pairs the start of the question slid out of the window before the answer was complete. On 29 February 2024 a four-epoch run of the 72M model finished in under nine minutes on my RTX 3070. Single runs lasted one to six epochs, and at least 14 continued from an earlier checkpoint.

In July and August 2024 I started a prototype for splitting one training job across several PCs: a Flask server that registered worker machines and sent them tasks, and a worker built on PyTorch’s DistributedDataParallel. I tested its parts on a toy model and never ran it at full scale.

The main training set is about 590,000 tokens against 72 million parameters: about 0.008 tokens of data per parameter, where a compute-optimal model would have around 20.

For an assistant, the test that counts is whether a request produces the right command. The eight models whose saved weights remain each answered 158 command requests that are in every version of the training set from January 2024 on, and 46 reworded requests that are in none. An answer counts only if the exact command follows <console>.

The models of 1 March and 25 May 2024 stayed the best: the one behind the chat window above writes the right command for 89% of the trained requests and 12 of the 46 reworded ones. The later models did not beat them; the 16-layer model of August 2025 gets 53% and 7 of 46. On a reworded request a miss is usually an answer with no command at all, and otherwise the wrong program: Photos or Edge for Photoshop, Chrome or Explorer for Excel.

The 0.8 confidence gate did not catch these misses. Run the way the chat window runs it, the 25 May model scored at least 0.85 on every one of the 204 requests, the wrong answers included, so none of them went to the web search.