Not having time for a person, not being able to sit in silence together with somebody, that's the same as rejecting them, as being scornful about them.