There's no rule, except "use a dictionary." And usage is slowly changing over time. So I'm going to check a dictionary, and I'll use my favorite, ahdictionary.com (which is US-oriented). nu·cle·us: n. pl. nu·cle·i or nu·cle·us·es gen·ius: n. pl. gen·ius·es If you want a cultural or historical explanation, here it is. In the 1800s, wealthy people went to universities that required proficiency in Latin as an entrance requirement. They learned Latin in private schools before entering the university. Many of them read it easily and loved Latin literature and poetry. Furthermore, there was a tradition that Latin was the language of science. So the tradition in the 1800s was: if a word had been borrowed from Latin, the Latin plural was used. That tradition had faded and continues to fade. And a dictionary will tells you what is most commonly used today.

Interesting, I would say geniuses and nuclei but I don't know if there is some official rule somewhere 🤔