I understand that thermoelectric coolers are perhaps a quarter as efficient as heat pumps. Why? What is the source of this efficiency limit? What would have to change for thermoelectric coolers to be more efficient?
The Peltier effect happens between two junctions of electrically conductive materials with different atomic lattice structures. Flow of current creates temperature difference between two junctions, but also generated the Joule heat in conductors. Also the link between two junctions is thermally conductive. So the thermolelectric element efficiency is limited by thermal "short" from cooled surface to hot surface and parasitic heat from dissipation in conductors. To increase element efficiency (aka "figure of merit", zT) the thermal conductance needs to be minimized, while electrical conductivity increased.
Unfortunately, Physics of thermal conductivity and electrical conductivity is bounded to the same internal mechanism - collective oscillations of atomic lattice, or phonons. Scholars figured out that the thermal conductivity has bottom limit for amorphous state of materials, which, unfortunately, has the lowest electrical conductivity. So the efficiency requirements for thermoelectric elements are inherently contradictory, and there is only so much one can do to improve the situation with band structure engineering using additions of different impurities.
Consumer peltier modules consist of P doped semiconductor connected in series (by wire) to an N doped semiconductor. This is daisy chained into a long string to be useful at a higher voltage. The majority carriers in each section are pushed in one direction by an electric field to create a temperature gradient. The majority carriers being electrons or holes are physically carrying their own heat from one side to the other. The inefficiency should becomes obvious when you realize that while the electrons and holes are travelling towards one side, phonons (lattice vibrations) can travel in the opposite direction. So while you're trying to move heat using the free carriers, phonons are constantly trying to make the entire item reach thermal equilibrium. Basically, you need to minimize thermal conductivity while increasing electrical conductivity.
Usually these two kinds of conductivity are quite related, but there are some unique materials that separated them. Carbon aerogel is one example of a material that may make for a good peltier device if it can be doped to have oppositely charged majority carriers.