There are solid state relays designed primarily for audio, which is what you'll want to use. There are in general some things you'll want to be aware of when using a solid-state relay.
First, you'll want to make sure the on resistance is compatible with your load. For example, if you are trying to drive a low impedance like a speaker, you cannot use a SSR which has high on-impedance (for example, 300 Ohms), as most of your power will end up being wasted by the SSR.
Second, you'll want to make sure the SSR actually has fairly flat frequency response up to the audio range (20 - 20k Hz). This will typically also depend on your load (that is, what you are driving), and generally looks better the higher impedance your load is (for example, if you are going into an amplifier versus a speaker). The datasheet will have all this information.
The third thing you must be aware of is that some SSRs are only designed to drive DC loads, which will not work for you (AC coupled audio signals go both positive and negative). A related issue is crossover distortion, which may distort your signal when the voltage amplitude gets close to the zero point. Again, characterizations of all this is available in the datasheet.
Your best bet is to find SSRs which are specifically specced for audio for example, this one: Vishay LH1518AAB