I have tried reading the SD card specification
The document will not give you much information about performance tuning - including block size effect on the performance. For compatibility reasons (and for some other reasons) use 512 byte block size.
My SD card is rated at 20MB/s
Rated for which operations - or is it maximal SPI communication speed? Let's do simple math: you drive card at 33 MHz, without considering the control overhead each clock cycle is one bit, thus to transfer a byte in SPI mode (single wire) you will need 8 clocks, thus 33/8 = 4,125 million bytes per second.
Source, page A-2.
Thus ideally SPI allows you transferring 1024 bytes in 248 us, which seem to be well below 1 ms.
However card's performance depends on some other things than SPI bus speed. It has its internal controller, which can be a bottleneck (sometimes - e.g. when CRC error or any other exception occurs, or always because it is slow), memory array with its timing can be a bottleneck. The actual speed of the cards can differ dramatically, thus you must dig into the specifications of the card deeper to find out read and write speeds, or better consult with manufacturer on what card may suit you and your pattern of access.
Example: Altera serial flash device I used recently (EPCS16) - same marking and same component source - earlier device erases sector within 1 second, later device erases it within 200 ms. This is dramatic performance improvement. The same for SD-cards - their timing may vary a lot. I also have heard that performance depends on the size of the card - bigger cards are slower, but I can not comment on it - just keep this possibility in mind.