Bigger phones are for women (and I guess men) who carry a purse. Otherwise it would be a pain in the ass unless you're some slouch with baggy and/or cargo pants.
That is going to be my only issue. I carry my phone in my front pocket. I already go without a case, because they make it too bulky and difficult to get out of my pocket.
So 4.7" may be the one I go with. I'll have to hold them before I know, though.