Well, because the street number is not a number //of something// (quantity), it really doesn't matter. The number has no meaning. I wouldn't say it is "very bad English" to say "one three five", but it is harder to remember. It's OK to read it as a number if you want.
I would say "one thirty-five" because it is easier to remember "1", "35" (This concept is called "chunking"). If your address has four (or more?!) digits, e.g. "1235", //nobody// will say "one thousand, two hundred and thirty-five" because it is too long. They will probably say, "twelve thirty-five". Again, it is not a quantity, so it doesn't matter. The same is true of phone numbers -- absolutely nobody reads them as quantities.
Obviously, if we ARE talking about quantity, you should say it as a full number.