You still need to read the file allocation table from the SD card unless:
you use your own RAW SD card format
Then you don't need the MISO pin. So instead of four pins for the SD card, you need three. It is possible to set a fuse in ATtiny85 that will
turn the reset pin to a regular I/O pin.
Now you have three pins you can use to talk to the SD card. You won't be able to use an ICSP programmer after you do that.
Share SD card pins with GPS.
Connect the RX pin with SCK. When you talk to the SD card, the GPS will receive garbage and will ignore it. When you talk to the GPS, keep the CS pin high and the SD card will ignore it. Connect the TX pin via a 10k resistor to the MOSI pin. When you talk to the SD card, use the MOSI pin as output. Communication won't get disrupted by the GPS, because it's behind the resistor. When you need to listen to the GPS, keep the MOSI pin as input and CS high. The SD card will ignore stuff coming from te GPS on that pin.
A combination of the above
Just use ATtiny84; it's almost the same as ATtiny85 with more pins and the same footprint in a QFN package anyway, if you care about size.