Your problem is that the plant you're controlling is high order. While non-linearity can be a complicating factor, it doesn't usually prevent stability once the time constants have been sorted out.
Let's put aside for the moment whether your PID electrical output is controlling the motor voltage or the motor current. Let's assume it's somehow magically controlling the propeller thrust.
The thrust provides acceleration, which integrates to velocity, which integrates to angle. So here you have two integrations, or 180 degrees phase shift. Any lags in the system will increase the phase shift further. So you have a system that will naturally want to be unstable. You are going to have to design stability into it.
Now consider your motor. I'd imagine you're controlling the voltage. There are several integrations yet to get to motor speed and so thrust, making the whole system higher order.
If the corner frequencies of your motor electrical 'input to thrust' co-incide with those of your 'thrust to angular position' system, then you have a very high order system that will be nigh on untameable. The trick is to move the two responses apart in frequency, a divide and conquer approach, often by speeding up one response and slowing down the other. A PID controller has only the D term to tame one order. It's usually not sufficient to cope with a high order system, you will need to design and add additional phase leads.
How to do that systematically, because with a high order system, hope'n'poke is almost always doomed to failure? Start by characterising the system open loop. Find the motor input that just makes it balance, then vary the motor input at various rates and measure the corresponding output variations. It can help for this stage of the process to reconfigure your see-saw so it's a pendulum. This guarantees the stability, and allows you to make measurements without it hitting the endstops, a form of nonlinearity that really messes things up. The extra stability of the pendulum is easily modelled, allowed for and removed.
Then hit google for what to do with this set of open-loop measurements. I usually use the Bode plot to see what's happening, but cleverer people just go straight in with Routh Horowitz. It's not easy, people do PhDs on and around these topics, but it is possible. And it does tend to need a mathematical systematic approach.
Finally, a note on non-linearity. It changes the gain of the system. As the position of the closed loop time constants depend on the total gain, they move about as the gain varies. A successful system design will place the time constants far enough apart that the system stays stable, even with their maximum excursion. Obviously, that needs a good understanding of where they are in the open loop.
