See I think that Simon's answer and Mara's response actually highlight the difference (confusion) between Am English and Br English.
In America "purse" is synonym for, or type of, handbag. In Britain it's not. "Purse" is the equivalent of a "wallet" in Br English in the sense that it's used to carry similar things, but a "wallet" is placed in a pocket and a "purse " is usually carried inside a handbag, because it's too big for a pocket. Since traditionally only women carried a handbag, a "purse" is typically owned by women not men.
Like Mara, I too use a bag in the summer to carry my wallet because I live in Italy and don't need to wear a coat... however I would never call by bag a "purse". Nor would I use the word"handbag". In Br English we call it a "manbag". It's still very rare to see a man carrying a handbag/manbag in the UK. Maybe this will change, as fashion does change, but at the moment a "purse" and a "handbag" are still predominately seen as for females, and therefore "wallets" are for men.