Voice is what we call "band limited". Meaning that it has a fairly well defined frequency band. A clap is essentially an "impulse" which is "broad-band" and thus has a very wide frequency range.
To put it another way, if you looked at the spectrum analysis of voice then you'd see stuff in the 200 Hz to about 12 KHz range. If you did the same for a hand clap then you'd see stuff from 100 Hz to beyond 20 KHz.
A hand clap also happens quickly, where voice goes on and on (and on and on and on :).
To build something that can distinguish between a clap and a loud voice then you would start by taking the audio and breaking it up into three or more frequency bands. Let's say <200 Hz, 400Hz to 8 KHz, and >13 KHz. You would then make a detector for each band which would detect peaks greater than some threshold, but not for more than about 200 ms. If you get a short peak on all three bands then you have detected a clap. If you don't get all three bands, or it is greater than 200 ms, then you have just loud voice.
Note: I'm doing an educated guess on the frequencies that I've mentioned here. They are approximately correct, but not 100% accurate. IF you want to do this approach I can kludge together a realistic test using real sound equipment-- to get the exact frequencies and time limits. It would take me about an hour to do, so I don't want to do it if you're not serious. But if you are serious, this could save you a lot of time (a lot more than an hour).