It is not unusual to have leakage currents through relay coils or transistors that are supposed to be fully OFF. The driving source may leak a few milli-amps of current, not enough to turn on the relay or a series indicator lamp.
12 volt relay coils are about 400 to 1,000 ohms of DC resistance, enough to turn off a conventional lamp but not an LED.
To fix the problem add a small value resistor across the LED such that the LED voltage drops to <= 2 volts down to 1 volt. That is below the 'ON' threshold of most LED's.
The leak is already there so a resistor to bypass the LED is not going to increase the leakage current. Remember there was a bulb there at one time, not impeding the flow of leakage current at all.
Measure the leak current if you can. A 1 K resistor creates a 1 volt drop across it per each mA of current flowing, so a 1 K 1/4 W resistor is a good starting point. It may have to be as low as 200 ohms if several mA of leakage current is present, but you said "dimly lit", which implies only 1 or 2 mA of leakage current.
NOTE: 'OFF' is not always an ideal OFF, as this case proves. I had to use diodes often to block leakage currents from electronic modules used to control overhead lights, and sometimes used to trigger a third-party alarm. Often it is a transistor that is not fully OFF simply because of a cheap design. It worked at the factory with standard light bulbs, not expecting LED replacements.