If you are powering that thing from a typical 9Volt battery, then of course you are going to have distortion.
3W Output at 10% THD with a 4 Ω load and 5V power supply
A 9Volt battery cannot supply any really large amount of current. They are typically used for loads under 100mA.
This guy ran some tests.
If you put a heavy load (more than 100mA) on a 9Volt battery, the voltage drops.
If you really load it, the voltage will drop so far that your 7805 can't regulate the voltage any more - the supply voltage to the amplifier will drop and you will get really horrible distortion.
To put out 3Watts at 5volts you are going to need 600mA of current.
See the chart for 500mA? At that current level, the voltage drops almost immediately to less than 8 Volts, and very soon is down to less than 7 Volts. The 7805 needs at least 2Volts between in and output, so when your battery gets down to less than 7 Volts it won't work right.
You need a different power source if you are trying to get 3Watts out of that amplifier. A 9Volt radio battery will NOT do it. You would do better to put several (4) AA batteries in series then use a low drop out linear regulator, or use several AA batteries in parallel then use a boost regulator to get 5Volts for the amplifier.
Added after comment about having tried other voltages and power sources:
The data sheet for the PAM8403 has a chart showing distortion as a function of power output and input voltage.
For 5Volts and 3Watt, you get 10% distortion.
For 3.3Volts and 1.5Watt you get 10% distortion.
If you put a heavier load on the amplifier, the distortion goes up.
To lower the distortion with that amp, you need to use the highest voltage you can and stay below 2Watts of output power. At five volts you will need to use a speaker with higher than 4Ohm impedance.
Another thing you may be missing is the gain.
The PAM8403 includes upto 24dB of gain. If you are using maximum gain (which you would be if there is no resistor between the bluetooth output and the amplifier input) then you will be amplifying the level by a factor of more than 10. Assuming a line level output from the bluetooth module, you will get really bad distortion playing at a normal volume. Test by putting a resistor in series with the inputs to the PAM8403.
If the problem is the gain, then you can set the gain by changing the input resistor. The total gain is 2*(142000/R) where R is the value of your series resistor.
Since you don't currently have an R at all, the amplifier is just running wild, amplifying for all it is worth - and distorting like crazy.
To get a gain of 1, use a series resistor of about 284KOhm (next standard size is 287K, which will do just fine.)
You also need to make sure to do all the stuff around it to make it work right