In American English, to ask a friend or acquaintance:
Would you like to come over for dinner (sometime)?
The "sometime" makes the invitation more vague and requires less of a committed answer from the person you are asking.
To be even more tentative, you can say:
How would you like to come over to dinner sometime?
It is generally understood that you mean your house, since it is usually rude to invite a person to someone else's house. But to be explicit, you can say:
Would you like to come over to my house for dinner?
Sometimes being explicit can help set the tone. For instance, the following is understood by native speakers to indicate informality:
Would you like to come over to my place for dinner?
Whereas, if you want to imitate the language of the 1960s, you could say
Would you like to come over to my pad for dinner? (But I do not suggest this!) Also, in American English we do not use the term *do you fancy?* This is British English.
You can also substitute the less formal word "supper" for "dinner" to make the statement even less formal.
For even a less formal way, used to ask a friend or someone you are getting to know:
How about you coming over (to my place) for dinner (sometime)?