Fine-Tuning Qwen 3 with LoRA on Apple MLX

Part 3 ended with two files: 364 training rows and 36 validation rows. This part turns them into a trained model.

The training itself is one command and 45 minutes. Everything interesting happened before it and after it.

What LoRA actually changes

The model has 4 billion numbers in it. Training all of them needs more memory than my Mac has.

LoRA freezes all 4 billion and trains a small set of new numbers beside them. The run reports it:

Trainable parameters: 0.182% (7.340M/4022.468M)

The result is a 29 MB adapter sitting next to a 2.3 GB model. The base model files are never touched. Delete the adapter and I have the original Qwen back.

The command

.venv/bin/python -m mlx_lm lora \
  --model models/qwen3-4b-instruct-4bit \
  --train \
  --data data \
  --fine-tune-type lora \
  --mask-prompt \
  --num-layers 16 \
  --batch-size 4 \
  --iters 400 \
  --learning-rate 1e-4 \
  --max-seq-length 512 \
  --steps-per-report 10 \
  --steps-per-eval 50 \
  --val-batches -1 \
  --save-every 100 \
  --adapter-path adapters \
  --seed 42

The flags that are decisions rather than defaults:

• --mask-prompt: score only the answer, not the diff. Without it, a large part of the run teaches the model to write git diffs.
• --iters 400: 364 rows at batch 4 is 91 steps per pass, so 400 steps is about 4.4 passes over the data.
• --learning-rate 1e-4: ten times the library default. The default is tuned for much bigger datasets and barely moves at this size.
• --max-seq-length 512: anything longer is cut off silently. I measured first: longest row is 361 tokens, so nothing gets truncated.
• --val-batches -1: score against all 36 validation rows. The default samples a few, and a sample of 36 is noise.
• --seed 42: same command, same adapter, every time.

--data data points at the folder, not the file. mlx-lm looks inside for train.jsonl and valid.jsonl by those exact names.

There is no epochs flag

This trainer only counts steps. How many times it reads the data is something you work out yourself:

passes = (iters × batch_size) ÷ rows
       = (400   × 4)          ÷ 364   =  4.4

So each row is seen about 4 or 5 times. Not 400 times. Change --batch-size and that number changes with it, even though --iters still says 400.

The run

45 minutes, 9.4 GB peak memory, a report every 10 steps.

the last 100 iterations

The loss:

iter    1    val 7.156
iter  100    val 0.488   ← lowest
iter  200    val 0.518
iter  300    val 0.554
iter  400    val 0.549    train 0.086

Train loss fell from 2.310 to 0.086. Val loss bottomed at step 100 and drifted up after. That is mild overfitting, and it matters later in this article.

The first result was a lie

The evaluation came back with empty messages and Marker [CG]: 0/14. That looks exactly like the adapter never loaded.

It had loaded. The raw output was fine:

'<think>\n\n</think>\n\nperf(currency): add caching for exchange rates [CG]'

Qwen3's chat template puts an empty thinking block in front of every assistant reply, including when it builds the training text. So all 364 rows taught the model to open with <think>, and it learned that perfectly.

Then clean_commit_message from Part 2 kept "the first line with content", which was now <think>, and threw the answer away. Ten lines of string handling written for a model that never produced thinking blocks.

The fix is one regex. The lesson is bigger: a zero that looks like failure is worth thirty seconds of printing the raw output. I nearly wrote "fine-tuning did not work" about a model that was answering correctly.

The scoreboard

Same 14 diffs, same greedy decoding, scored the same way.

One extra correct answer is a small gain. The prompt going from 800 tokens to 20 is not. The rules moved out of the prompt and into the weights, which is the entire reason Part 3 exists.

The marker is 0 before and 14 after. That is the plumbing check, and it is binary: the adapter is loaded and being used.

What actually changed

09-node-perf-cache → perf. Part 2 could never fix this one. I gave it a perf example using a set lookup, and the sample uses a cache. Different technique, so nothing carried over. Training crossed that gap.

13-kotlin-fix-lifecycle → fix, from feat. Another Part 2 failure.

docs(README) → docs(readme). Part 2 called this unfixable. Every rule about lowercase scopes lost to the fact that README is a filename. It does not show in the score, because docs was already right.

One regression: 14-dotnet-feat-pagination went from feat to perf. It was correct before training and is wrong now. Two fixed, one broken.

The best checkpoint was not the best model

--save-every 100 left four checkpoints behind, so I scored the one with the lowest validation loss against the final one:

The standard advice is to take the checkpoint with the best validation loss. Here that would have shipped the weaker model by three answers.

Loss measures how likely the model thinks the exact training wording is. It does not measure whether the commit type is right. They are related, not the same, and only one of them is the job.

Trying it

One flag switches the tool to the trained model. Same diff, same script, no adapter:

And with it:

--adapter-path switches three things at once: load the adapter, send the short training prompt, drop the few-shot examples. They are not separate flags on purpose. The model learned the shape of that exact conversation, and asking it any other way gives back part of what the training bought.

Fusing it into one model

Right now the tool loads two things: the base model, then the adapter on top. That works, but it means carrying two files and remembering to pair them.

Fusing does the addition once and writes ordinary weights:

.venv/bin/python -m mlx_lm fuse \
  --model models/qwen3-4b-instruct-4bit \
  --adapter-path adapters \
  --save-path models/qwen3-4b-commit-cg

• --model: the frozen base model, exactly as downloaded in Part 1.
• --adapter-path: the adapters/ folder the training run produced.
• --save-path: a new folder. Both inputs are left untouched.

It printed one line and finished.

models/

The new folder is 2.1 GB and holds the weights, tokenizer, config and chat template. There is no adapter file, because there is nothing left to attach.

Fusing is not lossless here

Scored the same way, the fused model matches: 12/14 types, 14/14 markers. But six of the fourteen messages are worded differently:

adapter: perf(currency): add caching for exchange rates [CG]
fused:   perf(currency): add caching for rate lookup [CG]

The base model is 4-bit. Fusing unpacks those weights, adds the adapter in floating point, then packs back to 4-bit. That rounding moves some weights enough to change a word.

I ran the fused evaluation twice and got identical results, so this is not randomness. Behaviour is preserved, exact text is not. Worth checking after you fuse rather than assuming.

What this cost

dataset          the part that took a week
training         45 minutes, one command
adapter          29 MB
fused model      2.1 GB, standalone
base model       unchanged on disk

The model is better at the job and needs a fraction of the prompt. It still does not write in my voice, for the reason Part 3 gave: the sentences in the dataset are the model's, only the types are mine.

I am not publishing this one. It ends every message with [CG], which makes it useful to exactly one person. That is worth saying out loud, because "fine-tune your own model" usually comes with an assumption that you will share it, and a model trained on your own habits is often the opposite of shareable.

The code

The training command, the fuse command, score_eval.py and every evaluation file in this article are in the same repository as Parts 2 and 3.

github.com/Charith1990/mlx-commit-lora

The adapter and the fused model are not in there, for the reason above. The dataset and the one-line command are, so the same model comes out the other end in about 45 minutes.

Next it goes into Swift, where a 2.1 GB local model has to behave like a normal part of an app.