Your understanding (higher flux = higher torque = lower speed) is not wrong, but it's based around a motor whose construction is reasonably close to the ideal, which is difficult to achieve with simple hand construction as seen in that video.
Now consider a motor with much less ideal construction, where bearings and brushes have significant friction, and flux is adjustable from low to very very low.
At "low" flux, the motor can turn at a few hundred RPM against the accumulated friction, but as you reduce the flux, yes the torque reduces, but instead of increasing speed, the friction simply stalls the motor.
So what can you do?
I would suggest using a much more "ideal" construction as a starting point for some more experiments. Toy motors can be found at several for a dollar, or even free in the right sort of old junk. (But at this stage, avoid anything that isn't a brushed motor) Bend a couple of metal tabs with pliers or screwdriver and you can pull them apart. If you lose a couple at that price, who cares?
You get accurately shaped magnets, curved to concentrate flux in the accurately made rotors, and good bearings. Mount those magnets on your stands as close as you can to that rotor.
Keep one set of magnets in the original can to see how the ring of iron around them both traps flux where you want it. Does adding a thicker soft iron ring outside the original preserve more flux?
Pull the wire off a couple of rotors and rewind with more turns, or fewer turns of fatter wire. (requires careful soldering to keep the commutator intact) What do more turns or fewer do to the rotor?
Measure voltage and current. If you can, measure RPM too.