Balloon Fit Doesn't Come Out
as Balloon Fit

Because they sit inside the average. Black sweatshirt fabric and elastic waistbands are everywhere on the internet. The model already knows them, so when you call for them, they show up.

The more extreme the shape, the fewer the examples. So the result gets pulled toward the closest thing the model does know. Ask for balloon and you get straight-leg; ask for ultra-wide and you get "reasonably wide." The middle photo above is exactly that kind of result.

That's the approach we just saw. It already fails on this very request: the request where we wrote "balloon" is the one that produced straight-leg pants. Words can't call up a shape the model doesn't know.

On top of that, nothing accumulates. Explain it a hundred times, and on the hundred-and-first you have to start over from scratch. You end up redoing, every time, something that fails every time.

We've got an open model, so why not just retrain the whole thing? That doesn't work either.

First come the practical walls. Retraining an entire model takes a matching amount of GPU power, every brand ends up with its own copy of a model weighing tens of GB, and while it learns new things, it forgets what it used to know. It learns balloon fit and then can't draw a shirt anymore.

The bigger problem, though, is how it gets taught.

Open up the entire model and teach it with a brand dataset of a few dozen garments, and the model won't bother looking for a rule. Memorizing those few dozen images outright is far easier.

It's like an exam covering 100 questions: if you can memorize every answer, nobody studies the underlying principles. It reproduces the garments in the training set perfectly, but with a garment it has never seen, it can't do anything. What we wanted was the rule "this brand uses this kind of curve." What we got was "that photo of those pants."

Here we flip the question. The images large models produce are, honestly, mostly good. The color, fabric, lighting, background and pose are all at a level we could never build ourselves. Only one thing goes off: the silhouette.

In that case, there's no reason to rebuild the model. All we need is this one thing.

Could we keep everything it draws well, and swap in just this brand's fit?

Without touching the model as a whole, you write down only "this brand uses this kind of curve" on the side, and add just that when the image is drawn. That's LoRA (Low-Rank Adaptation), and it's the only one of the three that works. In the next section we'll look at how that's possible.

Think of the model as a very thick manual on how to draw. Draw clothes like this, wrinkles like this, light like this: billions of lines of instructions like that. Rewriting all of it is the "retrain the whole thing" from the previous section, and that doesn't work.

So we don't change a single letter of the manual. Instead, we attach one thin correction note next to it. It's a sheet with only the differences on it, like "this brand's pants balloon out this much more, right here."

When drawing, the model reads the manual and the note together. It draws exactly by the manual, except for what the note says to do differently.

Before training, the note has nothing written on it. So at first, the result is exactly the same as the original. As training goes on, it fills in line by line, and how much has been written is how much has been learned. We use this value later to check whether training is going well.

When drawing, a single number sets how strongly the note is applied. This is called s (LoRA scale).

At s = 0 the note isn't read at all, so you get the original as is. At s = 1 it's applied exactly as written, and at s = 1.3 it's applied 1.3 times more strongly than written.

This is where the problem from the previous section turns around. If the note were as thick as the manual, it could copy down the training photos wholesale. And we'd be back to memorization.

If the note is thin, there's no room for that. It can't write down every one of the few dozen garments, so it ends up recording only what those garments have in common. That's exactly the "this brand's curve" we wanted.

Small isn't just cheap. Small is what makes it learn.

Think of the tuning feature in Google's Vertex AI. You upload your data, and they adjust their model to fit you. It's convenient, and the results are good.

But the tuned result stays on their servers. We can call that model and use it, but we can't take it with us.

We take a model with open weights, like the bottom three in the earlier table, and make the note on our own GPUs. It takes more work.

In return, the finished note file stays in our own storage. We choose it, we build it up, and we use it.

A brand's fit and silhouette rarely change. So per-brand adapters accumulate. The more garments come in, the better that brand's adapter gets, and the better it gets, the harder it is to move elsewhere. One file is one brand, so even at 100 brands there is still just one model.

What's hot now won't be hot in six months. So you have to train again and swap it out. With LoRA, training takes a few hours and the files are small, so swapping is realistic. If the setup required retraining the whole model, this axis would be flat-out impossible.

Draw without the note (s = 0) and you get the untrained original. Draw with it and you get the trained result.

That means you can generate "before training" and "after training" side by side, with the same garment, same background and same pose. Only one condition differs, so if there's a difference, it comes from the training.

Without this, even when the result looks good, you can't tell whether the training worked or that garment was just easy.

Because the note file is small, during training we save a copy of its intermediate state twenty times. A note that has learned a little, one that has learned about half, one that has learned all the way: twenty sheets in total.

When training is done, we try all twenty and pick the best one. There's no guarantee that the one trained to the very end is the best, and in practice there were cases where a middle one was better.

If we were retraining the whole model, we'd have to store twenty copies weighing tens of GB each, so this wouldn't be possible.

This sets how many pixels of each photo the model looks at while learning. Photos larger than this are scaled down. Looking at them small makes learning faster but smudges fine detail like the weave of the fabric; looking at them large is slower but picks up that detail too.

This is how many times the note gets revised. Think of it as the total amount of training. Too few and it learns less; more and it learns more, but past a certain point it actually gets worse (it starts memorizing).

This is how strongly the finished note is applied when drawing. Since it's set after training is done, you can try different values without retraining.

This is the thickness of the note. More lines let it record more, but too many let it copy down the training garments wholesale, so memorization gets easier. It has to be reasonably thin so that it only records what they have in common.

This is how big a change each revision of the note makes. Big changes learn fast but overshoot; small changes are stable but take a long time.

Show 155 garments 16 times each and you get 2,480 steps. Each step takes about 8 seconds, so 5 hours in total.

If the model learns from photos twice as large, each step takes longer. Once training is done, the time to generate one image also goes up from 90 seconds to 140 seconds.

When we first trained on small photos, the results were best at around 1,375 steps. So when we retrained with photos twice as large, we figured we'd just look around the same point.

But the results at that point were terrible. With larger photos there is more to take in at once, so at the same number of steps, the model had learned far less.

We kept it running, and it kept improving even past 2,000 steps. "Same number of steps, so it must have learned about the same" turned out to be wrong.