If the permanent magnet is a lump of iron then it is also a ferromagnetic core.
If it's magnetised so that it is saturated (off at one of the flat ends of the B-H curve) I'd expect its permeability to be lower than the unmagnetised core, and that will reduce its effect on the inductance.
Hmmm, that must be how those so-called "permeability tuners" worked - move a magnet closer to a ferrite coil, saturate the ferrite, tune your TV that way...
Lossier (because of that part-saturated core) than a straight capacitance-tuned circuit but it used to be popular in cheap consumer electronics in the 1960s? 70s?
Someone was looking for an alternative to a voltage variable capacitance just yesterday... maybe this (control DC current and hence magnetisation in a separate winding on the same core) might answer his purpose?