I am trying to create a cooling stage that can maintain 4 degrees Celsius however the stage must be optically transparent. Will a Peltier cooler be able to cool an acrylic sheet, or will the low thermal conductivity cause the Peltier to overheat without cooling?
Certainly depends on the size + mass (i.e. thermal mass) of your acryllic sheet -- not to mention the capability of your peltier.
In my experience, many peltier stages are surprisingly effective at cooling the area directly above their Tc face, but can have difficulty distributing that heat transfer to any material that is not directly in contact with the stage. Obviously, if you can find a way to use a material with greater thermal conductivity then you'll have much better luck.
However, like Jules said, without a sizeable heat sink you're pretty much guaranteed to toast your peltier as soon as you apply any noticeable amount of power.
Also, regarding your comment...
...cause the peltier to overheat without cooling?
... this will NOT happen so long as you: - have a heat sink on the Th side - do not have a large thermal load originating from the acryllic sheet
You can have a peltier sitting on your desk with nothing on the Tc side, run it at full power, and so long as the heat has somewhere to go your peltier will be perfectly fine (just an extra chilly Tc face).