I would like to get audio output through a pair of small desktop speakers (standard computer speakers - Logitech or similar). Is it safe to directly connect the ground bit of the 3.5 mm jack to the arduino ground (and same for some digital output)? Would any speakers just work (with/without their own amp)? I don't want either arduino or speakers to break.
Desktop speakers typically expect to be connected to a line level output. This means that you'll need to be able to produce both positive and negative signals around the ground, and have them on the order of several tenths of volt, not the 5V that's easy to produce as output from an Arduino. It'll need to be able to supply current in the range of several milliamps (it's hard to be sure exactly what the impedance of such speakers will be, so assume it's relatively low), too, so just dropping the voltage with a resistor divider isn't likely to work very well.
I suspect the easiest way of achieving this would be to use an audio amplifier IC (e.g. TL082, which contains a pair of amplifiers thus you could use it for stereo output if you wanted) connected to the output line via a capacitor (to avoid any DC coupling, only allowing AC signals to pass), and a resistor connecting between the output line and its ground reference (so the output line stays roughly around that ground level, whatever it may be -- it doesn't need to be the same ground you're using). Using one of the standard amplifier designs from the ICs datasheet, you'd then set the gain by experimentation to produce an appropriate voltage difference as measured by a multimeter before connecting to the speakers.
If you're planning on using PWM to drive the output, adding a low pass filter would be a good idea, too. As an alternative to PWM, you may want to use a resistor ladder, which is a simple way of producing an analog signal from digital input, and may produce higher quality output.