On my own boards, even the most dense, I never go below 25 mil height by 5 mil stroke, and I even hate to go that small because it's practically unreadable. Generally speaking a "safe" designator size is 45 mil height by 10 mil stroke. There will most definitely be cases where this is much too large, so I would say use a size as big as you possibly can, but no smaller than 25 mil by 5 mil.
When arranging designators on dense boards I sometimes find it useful to group designators together slightly away from their respective components, and draw rectangles and lines to indicate which "group" of components they go with. I generally do this for diodes, resistors, and capacitors. Sometimes the lines may not even be necessary, especially if your components are in a unique shape which you can imitate with the silkscreen (see below).
There are a number of tricks that help make positioning designators easier, even when they are relatively large. Whether or not to include lines "grouping" the designators and components together is completely up to you as the designer. Just keep in mind that clarity is key. If you don't make it clear which designators go with which components, your board house will keep coming back and asking you for clarification.