The correct instruction is "Keep silent". Equivalent phrases are "Stay silent" or "Remain silent", or "Keep quiet": always with an adjective. "Keep silence" is not natural in this context. I googled it and found a handful of examples, mainly archaic and/or religious. For example, in the bible quote "But remember what says the wise man, that there is a time for all things under the heaven; both a time to speak and a time to keep silence (Eccles. iii. 1–7). This sounds very literary and old-fashioned to the modern ear. It is sometimes used with reference to monasteries and convents where the monks or nuns 'keep silence.' I guess you could use "Keep silence", say, in the context of an examination room, where 'silence' is the rule that you have to keep to, but it is not common. 'Keep one's silence" also exists: this has the meaning of not to divulge a secret, as in "He swore that he would keep his silence". But even that is literary and outdated. My advice: forget about 'keep silence'.

Keep silence sounds weird and I agree that it isn't correct. It would sound more natural to say, "Please maintain silence during the exam". The command form was covered nicely by @La Liseuse. :)

In the classroom context you give, I would say "keep silent" - because at that moment in the lesson you want the students to concentrate. However in the context of a formal exam, where silence is expected for an extended period (i.e. the rule is to keep silence) - you might say "Keep silence" - this latter use being linked to the application of a rule.