I'm designing a home-made amplifier and am trying to choose connection options for the speaker outputs; is there any reason that RCA plugs are inappropriate, assuming I control the full system, including making the correct cables? Can they cause interference of some sort? They would be more compact than banana sockets or terminal connectors, but seem rare - is this purely convention or is there a motivating technical factor?
RCA connectors typically have a maximum current rating of 2 amps.
For an 8-ohm speaker, that's 32 watts.
EDIT - Per a comment by Dwayne Reid, I have found that Switchcraft makes RCA connectors which are rated for 6 amps (I'm impressed). Those puppies will handle nearly 300 watts at 8 ohms (200 watts RMS to be conservative). On the other hand, I've found RCAs that are only rated for 0.5 amps, which amounts to 2 watts. So, you can use RCAs for your speakers as long as you are absolutely sure of your source.
There is no reason not to use RCA connectors for (small power) amplified outputs.
Plenty of professional equipment uses RCA for line level signals, that is more liable to pick up noise, with no problem whatsoever. Maybe they use some sort of fancy connectors like gold plated but you know... Doesn't really make a difference in your project probably.
The maximum power is somewhat limited but probably the cable you are going to use is the biggest problem. RCA plugs can carry some 2A, that is plenty of current for a satellite of an home made amp, just be sure to use appropriately sized cables and you will experience no problem.
I would never use anything other than XLR for a live show. RCA has exposed grounds, which can cause a 60hz hummm if your power to everything isn't filtered and perfect.
For any other reason like making a home amplifier... just make sure your ohms rating can be handled by the line, as @WhatRoughBeast suggested.