One way this sort of peak/valley behaviour can happen with DRAM performance: there are several inter-related speed parameters that have to be satisfied.
One is the simple bus speed that you are adjusting : as you increase it, data transfer to/from DRAM gets faster.
Another is the internal access time between the 2-D memory array and a buffer that holds a single row of values (Row Access Time). Usually after fetching a row, we do a lot of accesses in that row (adjacent pixels share the row) so this has relatively little effect.
Another is the delay between asking for a particular value from that row (i.e. at a particular column address) and that value being available on the output pins (Column Access Time).
Now the problem comes because the Column Access time we need is a fixed value (say 20ns) but the bus speed is variable. But the internals can't usually work in fractions of a clock cycle, so Column Access time is actually expressed as a whole number of bus cycles; the datasheet may call this CAS Latency or CL.
Now if CL = 7 cycles at 1375 MHz but you then increase the bus speed a little, CL has to increase to 8 cycles to meet the Column Access time, so memory accesses slow down, and actual performance drops. Increase bus speed and performance increases again, until eventually 8 bus cycles is too short for Column Access time, and CL has to be increased to 9 cycles.
These are not exact numbers; you would have to find those from the datasheets on the memory fitted to your graphics card.
EDIT : playing with a spreadsheet :
You observed highest performance at 1375 MHz and 1650 MHz.
These frequencies happened to be 275 MHz apart.
Coincidentally (not!) they happen to be 5 * 275 MHz and 6 * 275 MHz.
Therefore your observations are consistent with a column access time of 1/275MHz, and CL=5 for frequencies up to 1375 MHz, and CL=6 between 1376 and 1650 MHz.
You can probably find a similar performance peak (and cooler running chips!) at 1100 MHz and CL=4.