I was assigned to observe the modulated signal of a PLC modem, but I have not really started yet. How do I do this? Should I use an oscilloscope or spectrum analyzer? If so, do I just connect it to the power line where the PLC modem is connected?
You didn't give the all-important information about what frequencies you are using for the communication. You can't know if a filter is right if you don't know what frequencies you want to separate.
However, the main point is to attenuate the power line frequency by a large factor. Here is one simple passive way to do that:
This is a simplistic filter which is basically 4 high pass filters strung together. I'm assuming the communication frequency is high compared to the power frequency, like 100 kHz or more. Each high pass filter stage has a rolloff of 1.6 kHz. 60 Hz is far enough below that so you can make the approximation it will be attenuated by 1.6 kHz / 60 Hz = 27. These filter stages will interact some to give more overall attenuation, but even ignoring that you get at least the single filter stage attenuation to the power of the number of filter stages, or about 495 k in this case. That means the 120 V 60 Hz power signal will be down to less than 240 µV out of this filter. 100 kHz, which we're assuming is your lowest frequency of interest, is 63 times the high pass filter rollof frequency, so won't be effected much by this filter.
The signal directly out of this filter is not ready for running into a chip like a opamp or microcontroller A/D yet. The power line has all kinds of nasty noise on it, with some transients that can be a few 100 V. This filter will pass such short term spikes, which would blow up a opamp or microcontroller. There needs to be some sort of clamping function, keeping in mind that those component have to be able to handle 100 V or more for a short time.
The voltage rating of the filter components needs to be carefully considered too. C1 will take most of the 60 Hz voltage, and certainly needs to be rated for power line operation. Even after just the first stage, the 60 Hz voltage will be low enough to not be a problem for ordinary resistors. However, the resistors need to be able to handle the temporary high voltage spikes that won't be attenuated by the filter. Either get resistors rated for a few 100 Volts, or implement the resistors shown in the schematic by 3 or 4 ordinary equal resistors in series.
You're saying little (nothing!) about the protocol. Is this Ethernet Over Powerline? The protocol matters because of the frequencies used.
Powerline communication both has a much higher frequency and much lower level than the 50/60Hz mains power. The higher frequency has its limitations: mains wiring isn't really fit for it, and the signal radiation will cause much attenuation over longer distances.
Instead of a transformer you can also insulate through capacitors. These have to be safety capacitors. Couple one capacitor from the phase and one from the neutral to the inputs of a difference amplifier. The capacitors will block most of the low frequency 50/60Hz power, but since the level of this low frequency is so high (120/230V) you'll need a notch filter to get rid of it.
A spectrum analyzer will not make you much wiser, you'll just see at what frequencies communication occurs. Which you probably know. A scope will allow you to watch the signal in the time domain.