This ideology becomes an issue when someone is finding ways to attack you that don’t quite constitute violence.
People need food, water, shelter, sense of belonging in society, etc. Bigots have gotten very good at using whatever means they can to attack each of these without ever physically throwing a punch; defunding someone’s means of living, evicting them, harassing them, etc.
Ideally, the law, and hence the police (who hold guns) would retaliate on each of these things.
I remember one where a sect of humanity was being persecuted, and left to try to find a better life in the stars. They failed to find a life, gave up and returned home, but their interstellar trip consumed so many Earth years that by the time they returned, Earth had moved on from persecution and eagerly welcomed their historical memories.
Sadly, I forget the name; it may have been a short story.