I think you could get enough energy from the sound card to trigger a fet and this could turn on a relay. Any sound card should be able to produce a frequency of 10kHz and a few schottky diodes and a couple of capacitors will generate enough voltage for a fet to turn on and activate a relay.
The relay could have to latch (using a spare contact to bypass the fet) and then the problem remains how to turn the alarm off but this can be solved with a manual reset switch. There are other options for disabling the latched relay using left and right audio channels but this may be over complicating stuff.
EDIT - simple audio to relay activator: -

The audio input (let's say 2Vp-p squarewave) gets level-shifted producing a +1.7Vpk and negative 0.3Vpk relative to 0V on the left-hand schottky diode. The right hand schottky diode and 100nF capacitor rectify this to produce a smooth dc level of about 1.4V.
Smaller input voltages will produce lower levels and I reckon with 1Vp-p input it won't produce enogh volts to drive a transistor. Choose schottky diodes with the lowest forward volt drop you can find.
The resultant dc level should be enough to activate the N channel MOSFET - again choose one with a low \$V_{GS(threshold)}\$ - maybe a BSH103 will do the job. I believe it will because the device should be able to activate a 50mA load with 1V on the gate relative to the source: -

50mA to activate a relay should rule-in most PCB mounted 12V relays and there will be less than 1V lost across tthe FET when activating it meaning 11V on the relay coil which should be more than enough. I'm labouring this point because the lower the relay current (EDIT) is, the lower your audio signal can be to activate it. Anyway, it's likely that Anindo Ghosh will point out an even better FET for this (he normally does!!).
The resistor is to discharge the 100nF capacitor across the FET gate and source should the "alarm condition" subside. 1Mohm and 100nF should result in a turn off delay of about 5*CR = 0.5 seconds. You might try and use a 100k resistor to turn it off faster but you'll be limiting the turn on time and possibly starting to eat in to how much voltage can be generated on the FET's gate.
Problems - You know have your sailing boat 0V wiring directly connected to your Mac-mini audio ground and this does concern me a bit. I'd consider two options here: -
- Isolating the audio signal with a tiny ferrite/Iron-powder transformer or...
- Using an optical isolator that has low forward voltage and produces an isolated voltage out for turning on a FET (just like used in solid state relays).
Photo-voltaic mosfet drivers are not uncommon these days and might be a better route. Here are a few from Dionics that I believe would work well. Most of them produce 6V or more with a LED forward current of 2mA. LED forward voltage is specified at 1.7V, 20mA so it should just about work.