What is this diode connected with the bypass capacitors? By googling 5.1V 500mW I came to conclusion that it is probably a Zener diode(however the symbol does not seem like a zener diode) Is that true? What is the use of it?
This is a Zener diode for over-voltage protection. Ordinarily, a diode conducts only in one direction. However, every diode has a "reverse breakdown" voltage, where the reverse bias voltage across the diode (or, the electric field across the PN junction) becomes great enough for the diode to start to conduct in the opposite direction.
Zener Diodes have finely tuned reverse breakdown voltages, and they can then be used to protect circuits from over-voltage by taking advantage of this reverse breakdown. If too much voltage is present, the Zener conducts in reverse bias. If designed properly, this will either (a) remove harmful voltage transients and allow the circuit to continue operating normally or (b) conduct enough current from the overvoltage source that a fuse blows, causing the system to fail safe.