One of the reasons why this happens, unfortunately, in my opinion, is very simple: as I wrote before, developers keep breaking the compatibility of their libraries with the rest of the world, whenever they feel, with no communication with downstream at all.
Since the amount of free software libraries is high, the probability of having a breakage is not something very uncommon as you may expect. Here in Sabayon we receive daily reports regarding things that break, that broke or could potentially break. We spend half of our (non-paid) time to just keep things working, without being able to fully dedicate to “progress”. With “progress” I mean, things that could make the Linux Desktop experience better every day.
A third reason for all this mess is human errors. At any level, and it’s part of the “domino effect” starting from developers mentioned in the first paragraph. Most FLOSS devs are very young boys and girls wanting to experiment and mess (in a good way, to make experience) with their code without being forced to follow an enterprise grade “code evolution path”, just to say one. We need more professionals, but to get there, we need to turn the free software thing into a profitable business, like it or not.
To make our philosophy successful, we need to find a good mixture of business and openness.
On The Other Hand 
I think requiring upstream to do what you seem to be proposing – maintain backward compatibility as much as possible, check changes against downstream, be bug free etc. would have an extreme cost unless it is almost all automated. And unlike one-off changes upstream, it would be ongoing cost for every release – I think it is pretty much unsustainable that way.
I think the only practical way to improve the situation is with automation. Downstream should be very easily able to see and follow up on upstream interface changes, and upstream should be able to build and run tests on massive amounts of downstream projects prior to finalizing a release, for starters… but we’re nowhere near this point.
Amen!
I’m so glad you posted this. It is a royal pain when developers don’t maintain backward compatibility and/or change a library major. People who package have to go back and rebuild everything against the new library then hope stuff still works.