The general rule of thumb with brushed DC machines is
Current ~= Torque
Voltage ~= (angular) Velocity
(to be fair almost all machines kind of follow this as well but it becomes less and less proportional and more "in some way related", eg freq)
You have two constants (kind of constants) when it comes to electrical machines
Kt & Ke
Ke is the open-terminal voltage constant with units: Volts/w. This produces a BackEMF
Kt is the torque constant with units: Nm/A
in theory Ke == Kt, but Kt is effected by iron characteristics (hence why two exist).
The Reason that Torque and Speed are said to be inversely proportional is the ability to generate torque diminishes with increase speed.
The reason for this is because the BackEMF opposes the supply that is attempting to force current into the stator, that will generate EM-Torque.
You are right that for a certain application of Torque a certain amount of acceleration will be generated based upon the rotor inertia and load inertia, BUT this torque will be reduced with increased speed as well (windage, bearings etc...). So between a diminishing ability to force current into a machine at increase speed as well as increased losses at higher speed the rate of acceleration will diminish until finally noload-speed is reached (or some loaded speed when compared to the load torque and the generated torque)