There are plenty of comparators that can be unofficially wired as an op-amp without much trouble but, many more will oscillate (go unstable) because they are not designed for linear applications; they are designed for non-negative feedback applications where output transistor switching speed is important. The LM339 is a bit of an exception in that you can get semi-decent op-amp style performance but you have to jump through a few hoops: -

However, speed is usually the big thing needed from comparators and that means the internally circuits are not compensated. And, without compensation, they will be often unstable when using negative feedback (op-amp applications).
But the control loop should be stable.
No, that's a much bigger and broader thing and you cannot make that conclusion at all.
Am I missing other huge downsides?
A lot of comparators don't have low input offset voltage accuracy but they will have (usually) a better common-mode signal handling reliability than op-amps (at the expense of worser input offset voltage specifications).