'Not really' is less emphatic than 'no'. We usually use 'not really' when there are elements of truth in what we are disagreeing with - to mean something like 'not exactly'. However, it's not always clear because sometimes, we use 'not really' instead of 'no' because it sounds more diplomatic.
Do you know your neighbours?
'Not really' .... could mean something like: I've met them (said 'hello' once or twice) but I don't know them.
Are you learning English?
'Not really' .... could mean something like: I watch English TV occasionally, but I'm not having lessons.
'Do you like Jane's new hairstyle?'
'Not really' .... could mean 'It's OK' or 'No, I hate it', or 'I don't know, I'm really not interested in people's hair'. Which is why this answer would probably be followed by another question 'What do you mean by 'not really'? :)