Great question. Actually, in this type of sentence "thought" and "expected" mean the same thing. And both can be followed by a phrase. You can say:"It was harder than I thought it would be." or "It was harder than I expected it to be." 都是一样。 As Eliot mentioned "it would be" or "it to be" can be left off. Hence, you can also say: "It was harder than I thought" or "It was harder than I expected" And all 4 sentences mean the same thing. Don't ask me why "thought" must be followed by "would" and "expected" must be followed by "to". Haha ... I am a bit puzzled by this myself. 有时候我觉得中文比英语容易一点。

than I thought = abbreviation for "than I thought [it would be]" You can say the long phrase or you can say the shorter one at any time, without any change in meaning. BTW, the long phrase has variations: ...than I though he would do ...than I thought he would have. ...than I thought she would be. I think you get the gist.

The tenses have to match