The following blog page Transmitter Fingerprinting contains some information. I couldn't find the particular patent mentioned although I was searching by the name Phil Farrell , possibly it was lodged under a company name or too old to be available in on-line patent databases. Certainly the claims made all sound perfectly plausible and relate to changes in the carrier frequency as the transmitter stabilizes. In that article it quotes the following as being from the former Motron website who used to sell the product:
The MoTron TxID Transmitter FingerPrinting hardware identifies individual transmitters using a patented technique based on the principle that carrier operated radio transmitters exhibit a unique frequency versus time start-up characteristic before stabilizing on the operating frequency. Carrier operated radio transmitters exhibit a unique frequency versus time start-up characteristic before stabilizing on the operating frequency - even radios of the same make and model. This 'FingerPrint' can be captured, stored and analyzed.
The TxID Software, which can automatically match and compare up to 256 FingerPrints, and the TxID-1 IBM/Compatible circuit board will help us to identify the abusers on the repeater. An onboard fast squelch starts the FingerPrinting process. The voltage on the receiver's discriminator is sampled, digitized and stored. The leading edge of the carrier is then captured, stored and displayed. Other information about the signal is also captured, including DTMF, CTCSS and DCS signals with separate peak deviation readings, and displayed with the FingerPrint.
The TxID System can optionally control a tape recorder, capturing all or part of the transmission on audio tape along with the digitally encoded FingerPrint data. Deviation measurements and Spectrum Occupancy features further enhance the system. The TxID System works with the Receiver, the TxID-1 can also capture the frequency of operation, as well as set the frequency.
There is also an open source project called XMIT_ID and while somewhat dated being written for DOS it does have source code available. I also found an article about a Windows application called SherlockXP written by WA9BVS although couldn't find a download location for it so I'm not sure if it's still available.
Those techniques are mainly aimed at transmitters with simple modulation schemes but I imagine for digital transmitters most of it would still apply and it could be augmented with further analysis in the digital domain.