There are downfalls with choosing very large resistors and very small resistors. These usually deal with the non-ideal behavior of components (namely Op-Amps), or other design requirements such as power and heat.
Small resistors means that you need a much higher current to provide the appropriate voltage drops for the Op-amp to work. Most op amps are able to provide 10's of mA's (see Op-amp datasheet for exact details). Even if the op-amp can provide many amps, there will be a lot of heat generated in the resistors, which may be problematic.
On the other hand large resistors run into two problems dealing with non-ideal behavior of the Op-Amp input terminals. Namely, the assumption is made that an ideal op-amp has infinite input impedance. Physics doesn't like infinities, and in reality there is some finite current flowing into the input terminals. It could be kind of large (few micro amps), or small (few picoamps), but it's not 0. This is called the Op-amps input bias current.
The problem is compounded because there are two input terminals, and there's nothing forcing these to have exactly the same input bias current. The difference is known as input offset current, and this is typically quite small compared to the input bias current. However, it will become problematic with very large resistance in a more annoying way than input bias currents (explained below).
Here's a circuit re-drawn to include these two effects. The op-amp here is assumed to be "ideal" (there are other non-ideal behaviors I'm ignoring here), and these non-ideal behaviors have been modeled with ideal sources.
simulate this circuit – Schematic created using CircuitLab
Notice that there is an additional resistor R2. In your case, R2 is very small (approaching zero), so a small resistance times a small bias current I2 is a very small voltage across R2.
However, notice that if R1 and R3 are very large, the current flowing into the inverting input is very small, on the same order as (or worse, smaller than) I1. This will throw off the gain your circuit will provide (I'll leave the mathematical derivation as an exercise to the reader :D)
All's not lost just because there's a large bias current though! Look what happens if you make R2 equal to R1||R3 (parallel combination): if I1 and I2 are very close to each other (low input offset current), you can negate out the effect of input bias current! However, this doesn't solve the issue with input offset current, and there are even more issues with how to handle drift.
There's not really a good way to counteract input offset current. You could measure individual parts, but parts drift with time. You're probably better off using a better part to begin with, and/or smaller resistors.
In summary: pick values in the middle-ish range. What this means is somewhat vague, you'll need to actually start picking parts, looking at datasheets, and deciding what is "good enough" for you. 10's of kohms might be a good starting place, but this is by no means universal. And there probably won't be 1 ideal value to pick usually. More than likely there will be a range of values which will all provide acceptable results. Then you'll have to decide which values to use based off of other parameters (for example, if you're using another value already, that might be a good choice so you can order in bulk and make it cheaper).