They really are the same, but the choice is between "rather" and "prefer to." You need the infinitive "to" when you use prefer, both positive and negative (that is, "prefer to" as an alternative to "prefer" or "prefer not to" as an alternative to "rather not").