RSA is not suitable for directly encrypting large amounts of data. Not only is it way too slow, but it is also weaker than some other algorithms against an opponent who has a large number of known plaintext/ciphertext pairs. To guard against this, data which is encrypted using RSA is usually padded with random bits beforehand. As a consequence, RSA ciphertext will generally be considerably larger than the size of the data payload.
The normal usage pattern for RSA is to generate a random key for some other cryptosystem and generate an RSA "cleartext block" which contains that random key along with some optional additional information about the file to be encrypted, filling any extra space with random bits. That block will then be encrypted and sent to the intended recipient along with a copy of the real file, encrypted using the other cryptosystem with the key that's included in the RSA-encrypted block.
I think that designing a serial "RSA accelerator" circuit to efficiently perform an NxK multiplication where K is a fixed value and N can be arbitrarily large might be a good challenge. Make it so that once the circuit is loaded with the short multiplicand, each chunk of the long multiplicand clocked in (LSB first) will generate the same-sized chunk of result data. If you use a chunk size of one bit, the multiplier could be implemented as N-bit two registers; if X is loaded with the K-bit value, and R with zero, then each step should perform:
if input is 1
output = R xor (input and K)
R = (R+K)>>1
output = R
R = (R>>1)
Implementing things that precise way would require a K-bit carry chain that could operate in one cycle. With a little thought, however, adding another K-bit register would allow the circuit to behave just like the above, but a rather short worst-case propagation delay. Further, the amount of additional circuitry required to tackle two bits at a time would not be much greater than that required to handle one bit at a time.
Designing a multiplier that could easily be interfaced to a microcontroller via one or two SPI ports (if two, make one master mode and make the other one slave mode with its clock tied to the master) might be a reasonable challenge. If one was willing to restrict the RSA modulus to the form 1000...000xxxxx with 256 zeroes near the leading bits (this would require using a modulus that was 256 bits longer than would otherwise be necessary for a given level of security) an Nx256 multiplier would probably require something around a 2048-gate FPGA but allow even a small micro to perform RSA encryptions in a fraction of a second.