You can't say that because "whole day" is neither the thing that you studied, nor how you studied. "I studied whole day" would mean you studied something called "whole day". You can say
"I stayed home and studied French" (what you studied. "French" is a noun.)
"I stayed home and studied hard" (how you studied. "Hard" is an adverb.)
"I stayed home and studied to pass the exam". (why you studied. "to pass the exam" is an adverbial phrase.)
You can also specify a period of time:
"I stayed home and studied three hours."
"I stayed home and studied between 2:00 and 5:00."
But "whole day" does not name a period of time. You would need to say
"A whole day"
"The whole day"
"for a whole day"