So we've failed EMC two times now because of spikes of 25 MHz harmonics from 175 MHz to 300 MHz, especially 175 MHz, 225 MHz and 275 MHz with peaks easily 30-40 dBuV above noise floor measured with Quasi-peak detector. We consulted every material on the topic we found, such as TI AN-1469 PHYTER Design & Layout Guide and we basically copied the layout from STM32 and DP83848 eval boards. We have what I believe is pretty standard design. STM32F407VET6 is connected to DP83848 via MII interface on 4 layer board.
We have identified the serial resistors on MII interface as a primary source of massive EMI radiation on mentioned frequencies (R11 to R20 in schematic). We can measure steady radiation from whole PCB, but around these resistors it is extremely high and we believe, it is then coupled to GND and / or VDD planes. We found only quick advices / urban legends around the internet about these resistors "that they should be there to minimise reflections on transmission lines" but we haven't found any thorough explanation or evidence about what exactly are they supposed to do and why they should be exactly 33 ohm.
EDIT: This answer sheds some light on the serial resistors, but not much.
Measured with oscilloscope, the RX_CLK and TX_CLK look like triangle waves from 0 to 3.3 VDC, so I accuse them of being the primary source of EMI radiation on mentioned harmonics. We will be replacing the serial resistors with Ferrite beads (BLM18BB331SN1) to see if it helps, but I would appreciate any info from anyone who dealt with this before and how he solved it. I also would like to know, whether there is something fundamentally wrong with our layout, as TI boasts about great EMC/EMI performance of DP83848, or if this is expected behaviour. We haven't found any substantial radiation from any other part of the PCB (other than GND / VCC planes, the peaks of EMI radiation have almost same amplitude on all parts of PCB where planes are, they only go much higher around MII interface serial resistors).
R51 is not used. We tried removing R50 & R22 and bypassing resistors on MII interface, but it didn't help much.
Schematic of PHY (caps on power supply pins are on different list)
Spectrum between 30 MHz and 300 MHz measured with near field probe.
EDIT 2: I marked answer by @analogsystemsrf as a correct one, as he indeed explains a lot of practices, that you have to follow in this kind of design. However I still think that this kind of design (MII or RMII without shielded enclosure) will be probably not impossible, but certainly difficult. My opinion is based on fact, that even EMC reference designs by TI such as this and this would not pass 30dBuV level EMI radiation test.
We indeed identified TX_CLK and RX_CLK wires of MII interface to be a main problem. We replaced resistors with BLM18BB750SN1 ferrite beads, which helped in range above 200 MHz, but harmonics in range 100-200 MHz started resonating, so this approach could work, but we would have to experiment with several ferrite beads. We also tried BLM18BB331SN1, but impedance was too high, which resulted in too damped signal and logical levels could no longer be detected properly.
Spectrum with BLM18BB750SN1 used on MII clocks
Shielding area of MII interface and side of PCB where GND 'fence' around PCB is broken by long wires by aluminium tape helped reducing radiated EMI by additional few dBuV, but we tested it only in one direction with near field probes.
Spectrum with aluminium shielding
Overall, I would recommend following guidelines mentioned in answers, mainly 'no gaping holes' in GND planes, keep MII or RMII clock signals as short as possible, proper impedance matching, ideally on the same layer without any changing layers and if it is necessary, provide current return path. Also have no other traces (especially long) too close to high speed clock signals, 25 MHz clock is indeed coupled significantly to first three traces above R11 resistor in our design.
We will probably 'solve' our problem in this design by reducing speed from 100 Mb/s to 10 Mb/s, as this mode only uses 2.5 MHz clock instead of 25 MHz and we don't need anywhere near 100 Mb/s speed for this device anyway. I will add spectrum measurement results when I have them.