I suppose I view patents as such harmful. Not just "software patents". Patents no longer serve the purpose for which the system was created. The content does not matter any more as much as quantity. Patents have become analogous to nuclear weapons: big parties use them to threaten each other and keep small ones out of their playground. The world has got over cold war - I think the technology industry should be able to get past the patent-era. We might need some mechanisms to "incentify" innovation, but patents should be banned altogether.
One central problem in the patent system is the fact that it is impossible to unambiguously decide whether something violates a patent. There is no universal definition of equivalence for abstract systems. And that is just how systems involving sofware are viewed: abstract systems - they are not viewed as their physical incarnation (if patent violation was decided according to bitwise comparison, avoiding patent violation would be trivial). This makes patent system kinda like a speed limit sign which would say "do not drive too fast" and the "too fast" would be left for a court to decide. So to avoid problems one could avoid driving. But what if one likes driving? What if one likes programming?
The problem with patents can also be viewed as namespace pollution. Imagine a programming language where anyone could register reserved keywords for money and users of reserved keywords would need to pay license fees. How would you like to program in such a language? But patents are much worse than registered keywords. Avoiding known keywords would be easy compared to the difficulty of avoiding doing something that could be covered by a patent. Patents pollute the namespace of creativity.