I would suggest the following two expressions. 1) Use "～～するその人" The person with whom I spoke was angry 　「私と話したその人は怒っていた」 (past tense) 私が話したその人　is also OK. The difference lying here is something negligible like the difference between "the person with whom I spoke" and "the person to whom I spoke". The reason for using その here is to clarify that you spoke with a person and that particular person was angry. If you don't put その here, it might sound like you are saying "People with whom I spoke became angry" because Japanese does not have the definite article (i.e., the) and does not usually distinguish singular and plural nouns. However, if you want to say "The people with whom I spoke were angry" in Japanese, you can say 「私と話したその人たちは怒っていた」. 2) Use a noun "相手（あいて）" when appropriate "相手" literally means a counterpart, an opponent, a partner, etc. Accordingly, you can change the sentence cited above into: 「私の話し相手(lit. my discussion partner)は怒っていた」　or 「私が話した相手（＝その人）は怒っていた」, etc. At the same time, this is a good solution to translate the second example that you posted earlier. For example, "I really liked the person I spoke with" can be translated into: 「私はその話し相手のことがとても気に入った（or 大好きになった）」