A switch where there are ≥2 “inputs” and ≥2 “outputs” and there can be simultaneous connections from any input to any output is called a crossbar switch. A quick search for “RF crossbar switch” suggest that they do exist as products, but are sold at the “if you have to ask, you can't afford it” level, which I assume you're not looking for based on your ham-radio tag.
If you were to build your own, the simplest approach would be to just construct the obvious electrical paths (two SP4T switches and four T-junctions), out of manual switches, relays, or electronic switching (e.g. PIN diodes).
The key part of such construction is to keep the length of all components with the improper impedance (in particular the inactive stubs of the T-junctions) much smaller than the shortest wavelength of interest, to avoid undesired reflections.
If you were to use four SPDT switches instead of passive T-junctions, and the switches themselves had the proper impedance, then you would avoid having any unwanted stubs and get better performance (limited by the specs of the switches). This increases the cost due to needing more switches, and also requires control logic to set the extra switches consistently.
Another feature you might want is to either short or terminate the two-of-four unconnected ports. This might require additional switching (I'm not sure exactly how much but it should be a simple “logic puzzle” to work out) unless you can get it as a feature in your SP4T switches, which you might if they're designed for RF switching.
Finally, I'd like to note (in case you're not already aware) that if you want to experiment with the possible topologies of switch networks without putting together a custom PCB design, that there are such things as coaxial relays, which typically come with SMA ports and terminals for the DC control signals, so all you need to add is some jumpers, a power source, and regular switches for control.
Of course, they're going to be more expensive than board-mount switches, but I've seen a lot of them available as surplus/used.