Yes, there's a logic. You'd say 'nip OUT to the shop' if you are INDOORS at the time of speaking; you'd probably say 'nip UP to the shop' if you are physically LOWER than the said shop. If there's a slope in the road, in either direction, you might say 'nip up to the shop' or 'nip down to the shop', as appropriate. I live on a steep hill, and I generally use 'verb + up/down + to' to talk about going anywhere from my house.
If you're in a flat on a floor higher than street level, you'd say 'nip down to the shop' if there's a particular shop close by that you'd need to descend stairs in order to reach. As Craig says, 'up' and 'down' can even refer to points on a compass, with 'down' for southerly movements, 'up' for northerly ones, and 'over' for east/west journeys.
If I heard someone say 'I'm just going to nip up to the shop', this would certainly suggest that the shop was at a physically higher elevation. Otherwise, I don't think that you'd say 'up'.
(This isn't necessarily the case with 'down'. We sometimes do use 'down' with no specific reference to relative positions. In informal spoken (British) English, we can even use 'down' as a kind of preposition to refer to regular places that you visit in your neighbourhood, as in 'I went down the shops' or 'I saw Jim down the pub last night' . I wouldn't recommend you copy this, but don't be surprised if you hear it.)
Are you correct thinking that you can't say "to nip out to the _store(s)_"?
Well, it sounds odd, because it's a mixture of styles. Using 'nip ...to' ( or the even more useful 'pop') is a very British way of speaking. It's a kind of diminutive, and is a way of reassuring the listener that whatever you're going to do is really, really, no trouble and it won't take long at all. American English speakers don't do this, so the nip/pop idea doesn't go with the word 'store'. A British person would 'nip down to the shop', whereas an American might say.....'swing by the store', perhaps?