I would say that the 1st one isn't grammatically correct - at least it doesn't make sense. I had been doing something for [a period] when ... tells me how long the thing had been happening before the point in time 'when'. When I was a student describes a period of time and not a point in time. So it would mean something like: I had been doing some thing for a time before I was doing something at the same time. - So it's a contradiction. In general, you use past perfect continuous with an action rather than a state. So when I 'started' or 'finished' something, rather than when I 'was' something. You might see it used with 'was' in passive voice, though. I had been speaking for 10 minutes when I was interrupted by the fire alarm. The second sentence makes perfect sense and you could also use past continuous: I was living in the town for 3 years when I was a student. I was living in the town for three years while I was studying. I lived in the town for three years while I was studying. You can't make the same meaning with past perfect continuous but you could use it in sentences like this: I had been living in the town for 3 years when I started studying. I had been living in the town for three years when I finished studying.

Although both sentences are grammatically correct, the second is much more logical. Let me explain why. We use the Simple Past (lived/was) to refer to 2 actions that happened at the same time in the past (e.g., 'I studied geography when I was at university' - they happened at the same time). So in the 2nd sentence you are saying that you lived in the small town at the same time as being a student. this is logical. The university was in the town, so you lived there. In the 1st sentence you are using the Past Perfect Continuous ('had been living') to refer to something that happens BEFORE the simple past event ('I was a student'). Another example would be 'I had been studying English for 5 years before I took the IELTS exam'. The studying happens BEFORE you took the exam. So sentence number 1 is theoretically possible, but it doesn't sound logical. We can also use Past Perfect to talk about an earlier event (e.g., 'I had lived' or 'I had studied'). The only difference is that the Past Perfect Continuous emphasizes the length of time - it tells the other person that you think the length of time is particularly important (maybe it's too long or too short). i hope that all makes sense :)

Both sentences are correct, but it depends on what you are trying to say. The second sentence is used very often and is used to explain that you used to live somewhere in the past. The sentence can be used on its own to make a statement of fact. The first sentence is used when describing an event in the past and something would be expected to follow this statement. For example, 'I had been living in that small town for three years when I was a student. Then, things started to change...' I cannot explain the grammar as I am not a tutor, but I believe the words 'I lived' are 'past perfect', a completed action in the past. The words 'I had been living' refer to an ongoing action in the past that occurs when someone is telling a story or describing an event. I do not know the name of this tense, but maybe a tutor will answer this.