"Look out the window" is more appropropriate here. The description is from the point of view of witnessing the event from within the classroom. If the witness was observing the event from outside the building you could say, "A few students were looking out of the window". Other examples: I looked out the window in time to see a man running across the lawn. I thought the house was abandoned until I saw a man looking out of the window.

Both can be correct. Usually, people would say "look out" the window. You can also say "look out THROUGH the window".

Example: He looked out of the window. This means he was on one side of the glass window and he looked through the glass.