I usually buy my pants about 2" bigger then what I can comfortablly wear, and belt them down. I just like loose pants and shorts...
However, I find that the looser the pants are, the less detectable an IWB is from the outside... just my findings... then again, i'm not a skinny person, so anything that I have on me tends to poke out a little (jokes aside)... you can clearly see if i have my cell phone or my cig pack in my pockets, and if I add a lighter or something else, it tends to show through, also.
ETA: body size does tend to play a factor... fat people usually cause their clothes to stretch more, showing signs of stuff below, where as skinny people... the clothes drape on them. Then again, at 6' tall, when I was 197, I still wore a 36 to a 38... and an XL shirt. I'm just a big person in general...