I'm planning on setting up a voltage comparator with a single rail op amp and was wondering if I still need to attach a diode to the output because it is single rail and not dual. I would think that the op amp will only produce an output when the input voltage is above the reference voltage not below it since it's a single rail. Is this correct?
Yes, that is correct. Usually a single supply op-amp will saturate quite near the negative rail. If the negative rail is ground, then it cannot output a voltage less than ground by itself.
Not all op-amps are suitable for use as comparators. Some, for example, effectively have diodes between the inputs. Recovery from saturation may also be rather slow.
I would think that the op amp will only produce an output when the input voltage is above the reference voltage not below it since it's a single rail. Is this correct?
No that is incorrect - an op-amp will produce an output voltage under ALL circumstances (except when burnt). On the other hand, a comparator might (usually or not) have an open collector output and when "open" the output voltage is not defined.
On a slightly different topic, there is no such thing as a single rail op-amp. Op-amps do not have a 0V connection they just do their stuff based on the two inputs and do not understand mid-rail or 0V.