There are many things that could be wrong between the oscillator and however you measured the delay you got. First, do the same thing with the known good 4MHz oscillator and make sure you get the expected measurement. Then try with the suspect oscillator setup and see what you get.
I wouldn't be using C and some possibly suspect delay routine to measure the oscillator rate anyway. Occasionally I want to check that the oscillator is set up as expected, especially on PICs with complicated PLLs and dividers. What I usually do is write loop that toggles a port pin every iteration, something like:
loop
btg portb, 0
bra loop
That loop takes 3 instruction cycles per iteration, so 6 instruction cycles per output period, and 24 oscillator cycles per output period. Now look at the square wave with a scope or frequency meter and multiply by 24 to get the PIC oscillator frequency after whatever PLLs and dividers are applied.