You're not alone if you're frustrated. below is a detailed explanation of what's going on with this chip, but it's a long read, so I'll stack some links at the top here to get you going in a hurry:
you'll probably get the best results by porting an existing library, or at least borrowing logic from some of the functions. Invensense recently released source code for version 6.12 of their MotionApps driver, and they also provide (only) the latest versions of their docs for the I2C/Register-based portion of their drivers. This is all available on their site after registering. Sadly, a lot of DMP functionality is not fully included... see below for more info on that.
I2CDevLib is a great source of info about the MPU6050. I have an open thread now in the hopes of stirring up discussion on how to activate the chip's advanced DMP features. Lots to read though, you'll need a long night to take it all in.
I've actually had most of my breakthroughs after scouring google for other MPU-based project discussions (quadcopters, cellphones, wii remotes etc). A lot of people use this chip, but for different reasons, so many forum users are unaware of one another's problems and solutions.
If you do search google, don't limit searches to "MPU6050" - There are several other units, like the 6500 and 9150, that share drivers. And don't be surprised if you reach page 10 of the google results - keep paging :)
So now you have a few pointers, here's the full story:
The code executed by the DMP co-processor on the MPU6050 is proprietary, and is not stored on-chip; the compiled version is included with each version of the Invensense MPU c/c++ files as a ~3kB array of raw bytes. To make matters worse, the registers and memory locations to enable or alter DMP functionality may be different for every version of the DMP firmware.
Even basic register maps and API documentation for features that are listed in product specs, like gesture detection and advanced sensor fusion, is hard to find. Some versions of Invensense's documents include sections that other revisions don't, alluding to functionality that is totally undocumented anywhere else. It's almost as though there's an "internal use only" version of their documentation, but members of the documentation staff occasionally include something confidential in a public release by accident.
My bet is that, unless someone reverse-engineers a binary, only Invensense will know exactly how the gears move in there.
Despite all this, there's a lot of promise in this little chip - Some of the constants in the .h files imply the existence of functionality that is very intriguing indeed, like Swing_xxx, Flick_xxx and many others. I recently managed to get Tap and Orientation detection data into the FIFO after porting a driver from some random release of the Android OS. Unfortunately, I found the source file after clicking through 20+ pages of google results, and I was so buried in Chrome-"new tabs" at the time I didn't even save a link ...sorry! Now that I'm getting data, I can see that Tap and Orientation only account for 8 bits of the 4 byte increase in FIFO-packet-size when "Send FIFO Gesture Data" is enabled. There's definitely more to be learned.
Good luck, let us all know if you make any progress!