Yes.
The voltage rating on a fuse tells you how high a voltage it can interrupt without arcing or otherwise allowing current to continue to flow after the fuse blows. A fuse of equal or higher voltage rating is fine as a replacement (after fixing whatever caused the old fuse to blow). DC applications require DC-rated fuses.
The conditions that will cause a fuse to blow are solely related to current (and time). A replacement fuse needs to match current rating and slow-blow or not; no more and no less, so it doesn't blow spuriously and doesn't allow more current than desired without blowing. A slow-blow fuse will allow some specified over-current for some guaranteed amount of time without blowing; useful for cases with high startup current.
All fuses depend on time, though, and slight over-currents might take a long time to blow, if ever. (The 4A rating is I think the max current at which it's guaranteed not to blow given unlimited time; vendors provide graphs of current vs. time to blow with some uncertainty range I think; some other fuse answers have included examples. Something like 4.1 amps in a 4A fuse might have an expected time-to-blow range that includes multiple minutes and never.)
Fuses are similar to a switch or relay that has to be able hold off a high voltage across itself. Interrupting a high current with an inductive load can create transient high voltages, which could create an arc. When a fuse first melts apart, you presumably do get some arcing over the tiny initial gap, but it presumably extinguishes quickly as more metal melts and the gap gets too long.
Fuses only rated for AC normally depend on the zero-crossings (every half-cycle) of the voltage/current to extinguish the arc, and aren't safe for DC usage, especially at higher voltages.