Substituting a 100 uF capacitor for a 1 uF capacitor is not valid at higher frequencies because the ESR and ESL of a tantalum or electrolytic capacitor come into play and become dominant at higher frequencies.
Stray inductance caused by layout can cause problems. Make sure these loops enclose a minimum of area:
- IC1 VB to C1 and back to IC1 VS
- IC1 HO to R1 to Q1 gate, and Q1 source back to IC1 VS
- IC1 LO to R2 to Q2 gate, and Q2 source back to IC1 COM
- HIN driver OUT to X3 pin 1 to IC1 HIN, and IC1 VSS back to HIN driver GND. Yes, you probably need to route a ground line with the signal; expecting GND to magically stay at 0 V via X1 pin 2 is probably not enough
- LIN driver OUT to X3 pin 1 to IC1 HIN, and IC1 VSS back to HIN driver GND.
Check VIH and VIL for IC1 HIN and make sure you're driving the right levels.
VC (Q1 drain) doesn't seem to be hooked up to a power supply.
Also, the 1N400x series diode might be too slow for the application. Try a Schottky or ultra-fast diode.
