Many of the languages that evolved from Latin (Italian, Portuguese, French, and Spanish) share two past tense forms whereas English only has one. Same problem with ser and estar (two verbs in Spanish, only one in English -- to be). In Fabi's examples, you can see that she didn't add a time reference in her second sentence. You cannot say "yo iba a tu casa ayer, el año pasado, en 2010" (specific point in time). However, you can say "yo iba a tu casa cada año, cuando era joven, durante el verano" (any clause not referring to one specific point in time).
That being said, you are right in commenting that both "fue" and "iba" mean "went", but try and think of "iba" as "(I) used to go". That should help make the difference between the two Spanish verb forms.
HTH