I almost always say "at breakfast." But, I'm referring to actually being AT breakfast, eating together.
If I were referring to the time itself, I would indeed say "at breakfast time." For some reason, though, I never say that.
I do say "at lunchtime" and "at diner time" quite often. I don't say it for "at coffee time."
Maybe that's because, for me, and for many Americans, "breakfast time" and "coffee time" are not well-established times. Each person has breakfast / coffee on their own schedule. However, "lunch time" is an established concept that refers to the 11:30 - 1:30 time range, and "dinner time" tends to refer to the 6pm - 8pm time range.
Hope this helps!