Is Villefranche-de-Rouergue safe for solo female travelers?

It's a nice city, but you can't ignore the suspicious people. You can meet very loud people in the street, sometimes there are fights too, and I've heard some (drunk?) men insult and threaten women in my neighbourhood (in the center of the city). I'd suggest to stay in the big streets at night if you're alone.