This is normal.
The multimeter tells you the voltage between the two input jacks. Nothing is connected to them, so, conceptually, nothing is causing the voltage to be anything in particular.
If the meter were an ideal voltmeter, you would see enormous numbers due to stray electrostatic effects.
Instead, it has a finite input impedance (which acts to discharge any applied voltage) and some leakage from the internal powered circuits to the inputs (which, working against the input impedance, causes the ~40 mV reading you are seeing).
If you short the terminals (touch probes together) you should see a reading much closer to 0. If you connect any voltage source that does not have an unreasonably high output impedance (more than a megohm), you should get a good reading.