*shrug* it's not just a size/weight thing, it's a displacement thing.
My V-Rod was only 1,130cc, about 120 hp, 70ish ft lbs of torque, but the minute you let out the clutch it wants to go. Compared to a smaller, lighter bike, when you're riding a V-Rod in the city, you're just constantly pulling in the reigns. For short trip stuff, smaller bikes are way more fun.
I don't see how this would matter to me in the least.
I've ridden big bikes, I've ridden small bikes. None of the variables between big and small would affect any of the decisions specific to what we've covered- I'd still dress the same, ride the same, and they'd both get me from A to B. I see only downsides in a smaller bike in the fact that they have way less storage for whatever you want/need to lug around during that "around town" stuff.