It just seems to be the way it works with most AVR chips. You could make your own reference on the board ofif you need a fast accurate one. Or you can look to see if the 100nF cap is across AREF as noted here: https://forum.arduino.cc/t/bug-with-analogreference/22994/19 which when removed brought the ADC set to internal 1.1V ref to ready in microseconds not milliseconds. Of course a decoupling cap is recommended for noise reduction (or a full lc setup), they ended up using a 10nF cap instead for half a millisecond ready time, and or using another pin to ground and drain the cap to make it ready faster as well.
That said you probably want to calibrate the internal reference as it's not fairly accurate in it of itself. Just fyi.