Your circuit is probably not working well because you are trying to drive a speaker which requires maybe 0.5 W with an OpAmp designed for low output currents. To give you 0.5 W across a 4 Ω load, you need an output current of I = sqrt(0.5 W / 4 Ω) = 0.35 A -- too much for a small-signal, general-purpose OpAmp.
Even with low-performance, low-current opamps like the 741 and its derivatives, it is possible to build a somewhat capable design, though. One trick is to boost the output stage by adding external transistors.
It will likely be easier to use a ready-made IC designed for loudspeaker amplification, but for the sake of learning, I'm posting a small experiment I've tried out a while ago.
The trick that can be learned is: What have the designers of ready-made loudspeaker ICs had in mind when they built their designs? What's the difference between an OpAmp and a power amplifier? They share a very similar cicuit symbol, so what's actually going on here?
If you look into commercially available audio gear, you might be surprised how often you find simple, general-purpose OpAmps, even inside of mid- to high-grade equipment.
Some clues and background info about boosting small-signal OpAmps can be found in an application note, originally published by National Semiconductor (now TI), and written by Jim Williams. (His name is not in the application note, but there's evidence ;-) Note that he even has some designs that use vacuum tubes to boost the OpAmps.
Here's a circuit I've tried a while ago, quickly drawn on the back of an envelope:

And here's what it does: It takes the OpAmp's output (741, half a 4559, whatvever), puts it through a complementary BJT output stage (Q1, Q2), and uses the BJTs' output signal as the input for its feedback, thereby eliminating any non-linearities and unknown properties (amplification = ?, linearity = ?) of the discrete output stage. Also, two additional transistors Q3 and Q4 are connected across the 22 Ω output resistors, provide an over-current protection that kicks in around Iout = 0.7 V / 22 Ω = 32 mA and protects the output transistors Q1 and Q2.
Output transistors are BC548/BC558. This design works well as a headphone amplifier. For loudspeakers, use bigger transistors for larger output currents and adjust the current sense resistors accordingly.
Note how the general idea looks very much like a basic, non-inverting OpAmp configuration, except for the additional BJT stage between the OpAmp's output pin and the circuit's output ("out") connected to the feedback resistor (22k).
The voltage of any signal taken from the input potentiometer (at the OpAmp's non-inverting input) will get amplified by a factor of A = 1 + R1 / R2. Q1 and Q2 just help to deliver more current while the OpAmp tries to maintain the desired voltage - and this is A Jolly Good ThingTM, quoting Mike Engelhardt, because for a speaker amp, you need both voltage and current, because power = voltage * current.
You may want to ask now just how much current the output stage is going to give you. Answer: The amp acts like a voltage source, and it will try to give you as much current as it can, just as long as Q3 or Q4 don't start to limit the show. Hence, the current is determined by the load connected across "out" and can be calculated once you know the load resistance RL: Iout = Uout / RL. Approximate a speaker's impedance using ZL = RL.