Welcome to the 1970s!
First, make sure you've got Dolby noise reduction on, if it's available. It boosts the treble when recording and turns it down on playback reducing the hiss and improving the S/N ratio.
If you don't have Dolby your best bet is to simulate it manually by increasing the treble during record, etc.
On my JVC tape deck I got good results by recording with Dolby and playback without.
Line out to mic in
Re-reading your question I suspect that you are feeding line output from your PC to mic input on your cassette recorder. If that's the case then the problem should be easily solved.
You have experimented and found that by reducing the signal level from Audacity that you can avoid distortion on the mic input. While this attenuates the signal it leaves the noise from the sound card output unattenuated. This means that your signal to noise ratio has decreased drastically.
simulate this circuit – Schematic created using CircuitLab
Figure 1. Stereo line to mono mic attenuator.
We can fix this simply. Send the signal out of the PC sound card at full volume. This will restore the signal to noise ratio. Then attenuate the signal and the noise between the PC and the mic input.
I would guess that the PC sound card will put out about 1 V p-p an that the mic input will be about 10 mV p-p. Try attenuating by 100:1 and see how you get on.
How it works
Take the left channel: R1 and R3 form a voltage divider given by the rule, \$output = input \cdot \frac {R3}{R1 + R3} = input \cdot \frac {100}{10000 + 100} = \frac {input}{100}\$ approx. It's called a 100:1 voltage divider.
The right channel works the same.
How to make
- Find some suitable resistors. Exact values isn't as important as the ratios between them. Cost is about 20c.
- Cut your 3.5 mm patch lead and identify the cores.
- Connect up the ends of the wires as per the schematic. If you don't have a soldering iron then use a screw terminal strip.