No. "With", like "by" and "beside" is a symmetric preposition. What this means is:
if A is with B, then B is with A
if A is by B, then B is by A
if A is beside B, then B is beside A.
This symmetry is an unusual property for a pronoun to have. I don't know if there are any other examples. "To" is definitely not symmetric. For this reason, "with" and "to" are not interchangeable.