An 8 bit shift register stores a byte of information. Would it be possible (in principle) to create a computer that uses shift registers for memory instead of HDD technology?
Yes, in fact the Kenbak-1 (considered to be the world's first personal computer, years before the Apple //) used serial memory. It was introduced in 1971 as a teaching machine.
It didn't use a microprocessor since the first one (Intel's 4004), had not been released yet when the Kenbak-1 was designed. Instead the CPU was made up of discrete TTL chips.
The Kenbak-1 had only 256 bytes of memory, implemented using two 1024-bit MOS shift registers. So it was rather slow, averaging only about 1000 instructions per second despite it's 1 MHz clock.
The instruction set was similar to the Motorola 6800, which would not appear until several years later. It had two 8-bit accumulators, A and B, and a 8-bit index register, X. The memory addressing modes were immediate (constant), memory, indirect, indexed, and indirect-indexed which provided more capability than most microprocessors introduced later in the decade.
Some of the early computer designs used mercury delay lines as memory. The process was that they transmitted acoustic waves into a long thin tube containing mercury which were picked up by transducers at the other and then retransmitted over again. It was invented in 1947 about 3 months before the transistor.
Before Integrated Circuits there were advantages of getting the data one bit at a time. You could build a very simple serial adder by shifting the numbers so that the bottom bits came in first. The sum will be shifted out at the same time, and the carry bit delayed one cycle to be added in. The RCA 1802 did this.