If VGA (video graphics array) is analog, then how is it compatible with a digital monitor?
It's not "compatible" by any meaning of the word. (I'd even say: a signal is never compatible with a device; a device might be able to deal with a signal). The digital monitor has an sampling input converter (an ADC) that converts the 3 analog pixel brightness signals (R, G and B) to 3 digitized sample streams, these get combined into pixels, which then are used to fill a sample buffer that is used to address the TFT matrix.
My guess is that maybe it carries an analog-modulated digital signal.
no, not in the way you mean it: there's no "image bits" exchanged between the graphics card and the screen, but three analog intensity signals. It's very archaic.
That's why VGA was basically a legacy solution since the late 1990s. Converting a digital data stream into the intensities in the device that actually contains the display, be it a cathode ray tube or an LCD screen, a plasma screen or an OLED matrix, makes more sense, but was prohibitively expensive for consumer screens until the early / mid 1990s. It's really a surprise you still find VGA, although rarely, on modern new devices, and a testament how formative the 1980s were for computing and displays.