I'm going to stick my neck out and disagree with everyone who has given their thoughts so far.
"I went to the library" - the emphasis is on what you did in the library, or the reasons why you went there, etc. We assume that, wherever you are, there is a library fairly close by, and no-one really cares which library it is. "I was in London the other day, and I had a spare fifteen minutes so I just popped into the library..." No-one knows which of the hundreds of libraries it was in London, and no-one cares. The topic of conversation is what you were doing in the library.
"I went to a library" - the emphasis is on that library itself and the fact I went in. Perhaps it was an interesting building (like the one in the picture), or they had an interesting event going on, or perhaps it was the first time I've ever been into a library.
I hope that helps. The two sentences are very different but it is quite difficult to pin down exactly what that difference is.