One way to access an SD card faster (up to 4x) than using the 1-bit serial SPI mode is to use the 4-bit parallel mode.
Unfortunately, this faster mode is proprietary to the SD Card Association. Just Google: "sd card 4-bit proprietary" and you will see a number of web sites that confirm this.
The SD Card Association has 1) not released the specifications for it (although some of it may have leaked), and 2) to legally use it in a product you must pay a hefty fee (a few thousand dollars). So even if you want to use it just for yourself, it is unlikely you will find a C library anywhere that supports this mode. But you can always look.
The SD Card Association has made the SPI protocol officially available which is way it is readily available, which you obviously are aware of.