"experiment" indicates that you're talking about just one. you could say, "I will do an experiment."
"experimentation" means that you're talking more about experimentation in general and could refer to more than one experiment. for example, "I will do experimentation." notice that you do not put an article like "an" before experimentation. does that answer your question?