(1) "Whose shirt is that" is correct.
(2)" Whose is this shirt" is also technically correct, but it sounds odd enough in modern American usage that my urge is to find a noun after "whose" to complete the sentence.(Note: You can definitely say "whose is better?" for "whose pie is better?")
Grammatically, "whose" is both a possessive adjective (explaining its use in (1) above) and a possessive pronoun (explaining its use in (2)).
Aside from the different use of "this" and "that" in your sentences, both mean the same thing. The asker desires to know "to whom belong (this/that) shirt."
However, I would definitely recommend saying (1) to sound more natural in everyday speech.