I heard that D+ and D- are differential signals, does it matter if I swap them when connecting a USB device to the computer?
When entering and exiting the idle state, the polarity is important and swapping the D+ and D- lines will cause problems.
USB data is NRZ-coded such that "One" is represented by no change in physical level, and "Zero" is represented by a change in physical level (see figure below). Therefore, inverting the signal (for example, by swapping D+ and D-) results in no functional change during data transmission. But there may be problems before and after data transmission which can kill communication with the device.
Exiting Idle State
The host includes 15 kΩ pull-down resistors on each data line. When no device is connected, this pulls both data lines low into the so-called "single-ended zero" state (SE0 in the USB documentation), and indicates a reset or disconnected connection. A USB device pulls one of the data lines high with a 1.5 kΩ resistor. This overpowers one of the pull-down resistors in the host and leaves the data lines in an idle state called "J". For USB 1.x, the choice of data line indicates of what signal rates the device is capable; full-bandwidth devices pull D+ high, while low-bandwidth devices pull D− high.
While the data is NRZI-encoded, the synchronization sequence and EoP are defined in terms of fixed states (J/K/SE0). When D+ and D- are switched, the J state is switched with K and SE0 is still SE0 (both lines low). So the sync sequence and EoP will become incorrect on inversion. In USB 1.x, if D+ and D- are swapped, a full-bandwidth devices get recognized as low-bandwidth and vice-versa. So the device will not even communicate at the same speed as the host.
Entering Idle State
A USB packet's end, called EOP (end-of-packet), is indicated by the transmitter driving 2 bit times of SE0 (D+ and D− both below max) and 1 bit time of J state. After this, the transmitter ceases to drive the D+/D− lines and the aforementioned pull up resistors hold it in the J (idle) state.
With a D+/D- swapped driver, the host will see the sequence (SE0, SE0, K) instead of the correct (SE0, SE0, J). The host might then fail to recognize the end of packet, which would cause problems.
If the device and host adhere strictly to USB specifications, swapping the D+ and D- pins will result in a failure. Its conceivable that the designer of the host foresaw such a failure mode, and built in compatibility for it. But whether or not such a swapped cable would be functional in practice, it certainly would not adhere to the specifications.
Another member, Andrew Kohlsmith, experienced this when the pins of a USB hub were accidentally swapped. The problem manifested itself as connected devices not showing up. The USB device would show it was powered but it was not recognized at all by the computer on the upstream side of the hub (which was wired correctly to the host).
Edit: thank you to those who commented. I added emphasis and details from your helpful notes.