Sir, the first variant is more correct. I suggest, "Can I ask you for a favor" or "May I ask you for a favor?"
The second variant can be adjusted to "Can I ask a favor of you?" When said this way, it is no more "relaxed" than the first option. Keep in mind, this is not Mr. Bean or Hugh Grant English; this is 100% Elvis Presley and Marilyn Monroe English.