Back in the day I lived in the outskirts of the city where buses would run rarely. Many times when I arrived home late a woman would also drop off. They mostly reacted in the same way as OP describes.

I tried to imagine having to be afraid like that all the time and it’s horrifying. The planet is populated by animals larger and stronger than you who might, or might not behave violently without provocation.