My first time building an AI model

ZeroLLM is my first model-building project. I wanted a writing editor where you could see which sentences read as machine-written, then work on the wording without losing what you meant.

I knew how to build an app around an API. This time, I wanted to understand what happened behind it: the data, the training, and how to tell whether a model was actually useful.

The code is AI-generated. I directed the experiments, chose what to test, and reviewed the results. These are my notes on that experience.

Two problems, two models

Marker detects patterns associated with AI-written text. Inkwell generates rewrites. They sit beside each other in the editor, but learning to classify writing and learning to produce it are different tasks.

I started from pretrained models and adapted them. I didn’t train a foundation model from scratch. For a first project on hardware I already owned, that made the work manageable and gave me something useful to learn from.

The build / explore each stage

From writing samples to the editor

01Data02Marker03Inkwell04Evaluate05Serve

Quality before quantity

Make the comparison fair

Check for shortcuts and overlap. A model should learn the task, not the formatting or subject differences between two piles of writing.

The data was harder than I expected

Finding human writing and AI writing wasn’t enough. I needed the comparison between them to be fair.

If the human examples cover one subject and the AI examples cover another, a detector can learn the subject difference. It might look good in a report and then fall apart on a real draft.

Related examples mattered too. An original document and several rewrites aren’t independent samples. Letting related versions cross between training and evaluation makes the test easier without making the model better.

I spent more time checking these shortcuts than I expected. Cleaning the data became part of building the model, not a task I could finish once and forget.

Getting the first run to work

My first detector training started on my Mac. I learned quickly that fitting the saved weights in memory doesn’t mean a training run will fit comfortably. Training has extra state to keep around.

Smaller batches and memory-saving techniques helped. Later, I moved experiments to a dedicated GPU, still working within a 16 GB memory limit.

The early detector result was encouraging. It also taught me to ask what a benchmark actually measured.

Marker / first run

What the first fine-tune changed

AI caughtAUROC

Share of AI examples detected at a cutoff allowing 1% false positives on human examples. Higher is better.

Starting checkpoint43%

After fine-tuning97.2%

That result came from a test close to the training distribution. It told me the adaptation had helped there; it didn’t establish the same performance on every kind of writing.

A rewrite has to keep the meaning

For Inkwell, I trained a smaller adapter around a pretrained language model. That made experimentation possible without updating every weight of the base model.

The harder part was defining a good rewrite. It needed to sound natural, but it also needed to preserve the writer’s meaning, facts, and intent.

A training target can look well written and still teach the wrong behavior. If it contains information missing from the input, the model is being rewarded for adding something the editor should preserve rather than invent.

That changed how I thought about data quality. “This is a plausible pair” was a much weaker standard than “this teaches the behavior I want.”

When newer wasn’t better

I initially expected another training run to move things forward. A later comparison challenged that assumption.

I used the same drafts and evaluation setup for the existing release and a new candidate, counting failed drafts in both results. The newer candidate did worse on the selected-rewrite measure.

Inkwell / matched evaluation

A newer run did not earn the upgrade

Selected rewriteFirst candidateUsable candidate

Drafts whose selected rewrite passed the enabled validity checks and received a human label from Marker 1.4.1. Higher is better.

Inkwell 1.1 · reference release45 / 80 · 56.3%

Inkwell 1.2 · candidate40 / 80 · 50%

This was a small test, so I couldn’t claim it settled every question. But it gave me no reason to replace the existing release.

Finishing a training run and improving the product are two different milestones.

A detector score wasn’t enough on its own either. I still needed to read the outputs, check what changed, and look for missing facts or awkward wording.

The model was only part of the product

A saved checkpoint still needed to work inside an editor: requests, waiting states, failures, and all.

I kept experiments separate from the release serving users. I also had to make long evaluations recover their progress when something disconnected. Neither changes the model’s weights, but both affect whether the system is usable.

What I’m taking into the next build

The biggest surprise was how much of the work happened around training.

The data determines what the model can learn. The evaluation determines whether an improvement is believable. The product determines whether that improvement helps someone.

I’m still testing and learning. I wouldn’t use a detector score as proof of who wrote something, and I don’t have a finished recipe for a great rewriter.

What I do have is a working path from data to an editor—and a much clearer idea of what to improve next.

You can try the project at zerollm.app.