What is your time scale? For peak detectors with any but the shortest hold times, I use FET input op-amps (hold times on the order of a second or more if you do everything right). Otherwise the bias currents will produce what you see. You can use a pot and add a small value to one of the inputs to zero the offset on an LM324. Or use a much bigger cap and an emitter follower before the cap to get higher current and fast charging. In precision circuits, use a polypropylene low leakage cap.
May need some capacitance across D2 for stability. Select C for speed.
(Are you really getting advice from that guy who spends 20 minutes testing design software without using any docs?)