They're both correct but would be used in different situations.
If someone is annoying they are doing something that you find irritating, tedious or frustrating. It could be important or trivial.
To be 'really testing your patience' they would be pushing you hard on something; this is stronger than being annoying.