As to efficiency concerns, other posters are right. You want to drive the Peltier with a constant DC voltage. To accomplish that we need some kind of switch mode power supply to supply that DC voltage.
Switching power supplies and Class D audio amplifiers are available cheaply in the target voltage and current range. They have the power handling capability to drive the Peltier, the question is how to get them to provide an electronically controlled DC voltage.
While the audio amplifiers have the right topology do drive the peltier in both directions, they would be harder to modify for DC operation. Take a look at this one for instance. Somewhere in there are some capacitors which if short circuited would turn it into a controllable DC voltage source. I personally wouldn't want to have to find them. The power supplies, with some external control circuitry are the easiest to make work.
Take a look at this module. It uses the LT3800 DC-DC step down controller. It's well suited for what I have in mind. The blue multi-turn trim pot near the input side. The one with the little brass screw you turn to change the output voltage. This terminal will be the feedback input.
The output voltage of the converter is fed to a resistor divider. The resulting feedback voltage is fed to the switching controller. The controller tries to keep the feedback voltage at 1.23 volts in the case of the LTC3800. IT does this by varying the PWM duty cycle. This produces a stable DC voltage at the output, which is what we want to drive the peltier.

Flip the board over and you'll see the trimpot terminals exposed. It's easy enough to trace out the layout and tell which one connects to the Vfb pin. when the converter is operating it'll have the reference voltage listed on the datasheet. Solder a wire onto that terminal and connect that to an external circuit through a resistor. This lets you vary the power supply voltage electronically. Be careful not to apply more than the rated absolute maximum(5v) to that pin though.
For driving in one direction, (Only heating/Only cooling), Connect one terminal of the Peltier to the power supply and the other to ground. The power supply acts as it was designed and provides a varying DC voltage which varies the amount of heating or cooling supplied. An external control circuit would be connected as described before. You'd just need an op amp some resistors and a potentiometer.
For driving the Peltier bidirectionally, you'll need two of them, Essentially, the control circuit would set one power supply at Vx and the other at (Vsupply-Vx). The converters would have to be modified since they're set to be non-reversible. To fix this, tie the burst enable pin to Vcc rather than Vfb (The manufacturer just copied the datasheet example layout). You can see it in the image on the Ebay page. The third and fourth IC pins from the right are shorted together, Cut the trace going to the Vburst pin and solder in a wire connecting it to the Vcc pin on the other side of the chip.

The control circuit would just be a 5v regulator, an op amp and some resistors, a capacitor, and a trimpot for the set point. 5V is chosen so the voltage on the converter feedback pin stays below the absolute maximum voltage.
Components are chosen to give the integral component a 10 second time constant.
The disadvantage with this is that you need to change the component values to tune the circuit. Using a microcontroller (EG:Arduino). to read the 4-20ma value, read a potentiometer to get the set point and produce an analog voltage would make the controller easier to tune. Just change the constants in the program. The code would be very simple and this would allow a computer to control the set point. Here's what that would look like:

You write some code for the Arduino. It reads the analog voltage coming in, converts that to a temperature, feeds that into a control loop and writes some analog value out that sets the voltage going to the peltier. You can even get another library to tune the control loop for you.