When people from the people business, stop caring ...

When people from the people business, stop caring about those who are genuinely concerned with not only about what you say, and how you say it. That's feedback.
Telling people off, that it's "your business and I don't why we're having this conversation" when you are dealing with a crisis, without the right damage control and shutting people down rudely, that's not right.
In hindsight, pick the right folk, have a trained team equipped for dealing with such concerns.
Find alternative communication channels and use them to leverage groups.
Don't be available 24x7 - Pick your work hours.
When you can't have difficult conversations, find someone equipped to handle that.
Somebody else's visual challenges should not make others verbally challenged.