I would say "I am happy to be with you"

I am happy to be with you -> I enjoy being together. I am happy with you -> I am pleased about something you did.