It's an old expression: " Fish and houseguests smell after three days". It's comparing houseguests to fish - after three days, they've been there long enough and they're over-staying their welcome. You get tired of smelling fish after three days and you want that smell gone; likewise, you want your visitors gone after three days.