Does anyone know some good resources (textbooks, papers, tools, etc) for developing a custom compiler and linker for a specific microcontroller?
The book "A Retargetable C Compiler: Design and Implementation" ( http://en.wikipedia.org/wiki/LCC_%28compiler%29 ) describes a C compiler in all its details. It uses the "literate programming" style, which I found more difficult to read than I expected. But I still recommend it.
Jears ago, before there were any suitable free C compilesr for PICs, I made the Jal language and a compiler for it for the PIC 12/14 bit cores. The source is GPLed (~ 10k lines C, in one file). Nowadays the language and compiler (still GPL, so you can study the source) is maintained by Kyle York ( http://www.casadeyork.com/jalv2/ ).
There is of course the GCC, but it is BIG. There are some master thesis documents that describe the re-targeting to a particular CPU, for instance http://lundqvist.dyndns.org/Publications/thesis95/ThorGCC.pdf , this could serve as a first step into understanding the full GCC internals (which would be a daunting task).
The are various academic textbooks about compiler building, the "dragon" book ( http://en.wikipedia.org/wiki/Compilers:_Principles,_Techniques,_and_Tools ) being the most famous. But IMO they are almost useless, concentrating on things that were difficult 20 years ago but almost irrelevant now.
You mention "compiler and linker". That shows a mindset that is IMO outdated. For most modern microcontrollers a modern PC can cope with the full application at once, bypassing the compiler/linker split. This makes much better optimization possible.