To me, it would mean brake off a portion of your life and cut yourself off from it. It would usually refer to a friend (or boy/girl friend) with whom you do not want to talk or associate with.
It might also be an activity or group that you were a part of and decide to separate yourself.
"When his friend at the bar started into drugs, he broke off that branch.
To "break off INTO a branch" would be to make a choice and follow a certain path among a number of different paths that were available.
To "branch off" means to separate yourself or your group from something you or they were a part of; or to go into some smaller area of a larger activity.
"From mathematics in general, he branched off into string theory."