Hello Emarbe,
I think here you need more context. Certainly 'arrange the party in such a short time' is correct if say for example it is a question. 'How can you arrange the party in such a short time?'
However there is another possibility which is 'He arranged the party at such short notice?' Note the change of preposition to 'at'. 'At short notice' means without being given a lot of time.
Hope this helps
Bob