In fairness, I would say that while the 'heavy-featured' is definitely problematic, 'big' is one of the things I'd assess in a physical confrontation - someone who is heavier and more muscular than me would feel threatening.

Then again, questioning assumptions further, this refers mostly to the kind of hitting/bodyslamming barfighting common in certain cultures; when you add weapons, martial arts (or, in fiction, magic) the odds can change drastically. Maybe it's the quietly confident guy with the grace of a dancer you need to watch, not the big lumbering bloke with slow reflexes.