Yes, you can. In fact, you should. It is far more natural to say 'As (we) said on the phone,.....'.
'As spoken on the phone' sounds wrong. 'Said' refers to the content of the phone conversation - for example, what you discussed and agreed on. 'Spoken' refers to the fact having spoken or the manner of speaking, so it would be very odd to say 'As spoken on the phone'.