My feeling is that "sit idle" and "lie idle" are generally used of objects.
In the lockdown, many office computers are sitting idle.
It does sound a bit strangle using it for yourself, I agree. You could say "ashamed of my idleness" or "of my failure to act", or many other things, depending on what emphasis you want.