It's hard for me to explain it.
You'd more likely say, "I'm very attracted by *the things that you say and do*, yes I'm very attracted to you".
I know the attraction is because of the person, so it's rational to think you would say, "I am very attracted by you" - but it's not good English.
"I'm very attracted TO you" not "by you".
To attract, is to make one thing come closer to another, if I have a ball of silver foil, my cat is attracted to it, if I wave some food at my Dad's dog, she is attracted to the food.
If I have a magnet and the polarity is right, the other magnet will be attracted to it -
To attract is to be at distance from a person, and for that person's looks or words or actions, to make you want to walk closer to them, you've been attracted -
So in theory I suppose you could say "I'm very attracted by you" - but no, in England we'd just say, "I'm very attracted to you" - it means the important part of the saying, is "TO".
You'd use "By", like; "I'm impressed BY the things you do" - "I'm disgusted by your actions" - "I'm hurt by your lack of consideration" (like me last night ooops stupid man!!)
But no, you'd not be attracted by a person, you'd be attracted by what they did or said, or looked like.
Whoever has attracted you is lucky :D
I hope I have helped and not made it more confusing.