You want pogo pins. Solder the non-pointy end into a board and push the receiving board onto the pointy end when you want to use it. Obviously you'll have some pattern of pogo pins and a matching pattern of smaller-than-the-pointy-bit vias (through holes) to mate with. The idea is they allow some wiggle room for mating boards, all the pins don't have to be perfectly the same length.
I use these for test boards. Make a board to match the test points on your device and build your test circuitry onto it. Recently I've started using them for very small programming connectors too, seems to be working pretty well.
The very small Spy-Bi-Wire programming connector I made looks like this on the board
This is the surface pad version on a 1 mm grid, I also made a through hole version of the same size for use if I have the board space on both sides (I suppose a blind via would work, if I didn't care about board cost). The programmer has an identical arrangement, but with through holes for the pogo pins to solder in (but pointing out the bottom of the board). The center is an alignment hole. Until I find a better solution, the programmer side of this connection has a thin pin where the alignment hole is, I stick that through the target board alignment hole and clip the far side so it doesn't slide back out. The surface pad version works fine if I'm not moving the board all around because it can accidentally rotate, though I've found the pogo pins usually stop at the solder mask (the through hole version doesn't have this problem). I'm still trying out other variations, but like I said, this one does seem to be working out quite well.