I'd just like to give a quick answer because I'm not good at these things, besides, my English is not suficient to explain, so please wait for better answers.
There are 番地 and 地番. As you said, 番地 is used for address of each house. On the other side, 地番 is for land, it's not for house which someone lives in, however, land owner is exist.