Depening on the context of the sentence, but in most cases, He was seen (past participle of see) entering the building-ing of enter is the present participle.
He was seen to enter the building does not mean he was with anyone. To always means direction, action or feeling. It can be infinitive but not always.