There are hundreds of different antennas so for my simple attempt at an answer I'll concentrate on the "dipole" and I'm not going to go into formulas too much.
What physical factor affects the bandwidth of an antenna?
The impedance presented by the antenna is a major factor. If that impedance changes with frequency, then transmitted power will also change. Consider this: -

As dipole length becomes about half a wavelength the impedance becomes real and about 70 ohms. But go to 1 wavelength and the real impedance becomes infinite. This is a severe factor affecting bandwidth. Given also that the imaginary impedance change is quite severe, using a physical impedance (a series inductor for instance) to allow effective operation at slightly off exactly one half wavelength will result in a significant tightening of the bandwidth.
So, a short answer is that bandwidth depends on how you use the antenna as well as the choice of antenna.
There is a useful video at antenna.com on this page that might give a bit more insight. The same website goes on to discuss how thickening the dipole wire can make bandwidth wider: -


It now talks about VSWR (which is still all about the impedance presented by the antenna) and shows that a thicker wire produces wider bandwidth and, if the analysis spectrum were widened you would see much more significant effects: -

Antenna.com's conclusion below this is: -
Hence, this page can be summed up very succinctly: to get wider
bandwidth, fatten up your antennas (this means: use more volume).
It's a really big subject but, for most regular antennas, the biggest decider of bandwidth is the driving electronics and impedance resonances. For instance, using a "short" dipole antenna antenna.com gives the following formulas and worked example: -




As an example, assume that the radius is 0.001 wavelength and the
length is 0.05 \$\lambda\$. Suppose further that this antenna is to
operate at f=3 MHz, and that the metal is copper, so that the
conductivity is 59,600,000 S/m.
The radiation resistance is calculated to be 0.49 Ohms. The loss
resistance is found to be 4.83 mOhms (milli-Ohms), which is
approximatley negligible when compared to the radiation resistance.
However, the reactance is 1695 Ohms, so that the input resistance is
Z=0.49 + j1695. Hence, this antenna would be very difficult to have
proper impedance matching. Even if the reactance could be properly
cancelled out, very little power would be delivered from a 50 Ohm
source to a 0.49 Ohm load.
For short dipole antennas that are smaller fractions of a wavelength,
the radiation resistance becomes smaller than the loss resistance, and
consequently this antenna can be very inefficient.
The bandwidth for short dipoles is difficult to define. The input
impedance varies wildly with frequency because of the reactance
component of the input impedance. Hence, these antennas are typically
used in narrowband applications.