Peter Bennett's got a point about the minimum and maximum voltages that can be present at the inputs, but I'm not certain it's the problem here. First off, we can look at the behaviour of your op-amp. With no input, it's being driven to the rails, as it's called. Basically, an op-amp's maximum range is it's positive and negative supply voltage, less a small amount, referred to as it's headroom. For modern op-amps, the headroom is usually less than a volt. For older op-amps, like the 741, the headroom required for the internal circuitry is much more. This should be specified on the specification sheet, if you're curious. I recommend http://www.datasheetcatalog.com/ as a source for that, although you can always get datasheets at the manufacturer's website. That describes the output voltage you're seeing from the op-amp. Rail voltage less some amount of headroom, likely a volt or so. It doesn't describe why you're seeing it, though. For this, you'll need to learn about how an op-amp actually works, and where a real op-amp differs from it's ideal model.
While I'm not certain of what your electronics background is, it seems like you're expecting the output of your op-amp to behave something like \$v^+ - v^- = v_{out}\$. That's an okay mental model of the op-amp, but it's missing one element, the gain. A better mathematical model is \$K(v^+ - v^-) = v_{out}\$, where K is a number. A really big number. Like, 100,000 or so (it's in the spec-sheet. Look for the Gain-Bandwidth Product). A real op-amp can't output at a voltage outside the range of it's supply voltage, so it's being driven to the rails hard. What you can do (and I'll add some schematics later) is add negative feedback to the circuit, which decreases the gain to a more manageable level. For a straight difference, you want the gain to be 1.
Finally, we want to know why the op-amp is outputting a voltage at all. For that, you need to know even more about the non-idealities of op-amps. Basically, everything inside that puppy is transistors. If I remember the 741 schematic, you're looking at 14 or so BJTs. I think wikipedia has an article that shows the schematic, if you're so inclined. At the input of the 741 is what's called a differential amplifier. This takes the difference between the two inputs, and outputs the result, where it's later buffered, most likely. The problem is that this takes at least four BJT transistors, and every transistor's going to be different. This creates small little dc biases in the differential amplifier, so even if you were to connect the \$v^+\$ and \$v^-\$ pins together, you would still see the op-amp driven to the rails. With a gain of 100,000, even 1 microvolt of difference is a big error.
As an aside, the 741 is an eight-pin package. Check out the spec-sheet, and you'll see that two of them, if I recall, are labelled null offset. These pins are actually used to slightly offset the differential amplifier, removing that error in the output. Newer op-amps are a lot more precise, and don't need to offer that functionality anymore. I still wouldn't recommend nulling the op-amp without reducing that gain first, though.
I'm going to hit you with something big.
simulate this circuit – Schematic created using CircuitLab
WHAM!
I stole that diagram from http://www.electronics-tutorials.ws/opamp/opamp_5.html . Give that whole string of articles a read, it's really great. This is an op-amp circuit that can be used to take the difference of two inputs. It's a bit of a handful to analyze if you don't have much of an electronics background. From a quick glance in my old textbooks, the output of that circuit is something on the order of $$v_{out} = \frac{v_2*R_1(R_3+R_4) - v_1*R_4(R_1+R_2)}{R_3(R_1+R_2)}$$ This is a fairly elementary circuit, and you should be able to find it easily in any university or college-level textbook, or online. Basically, if you want your subtraction operation with a gain of 1, just make all the resistors the same (I recommend 10K resistors). The equation reduces to \$v_{out} = v_2 - v_1\$.
