Being in position to make a game-winning play and failing is just about as bad as just plain failing. And it's what people remember more.
Ginn wasn't asked to make particularly amazing plays that he just didn't come through on. He was asked to make plays that are very basic for just about any NFL receiver, especially a starting one. In the Saints game, he failed to come through on even more basic catches and on top of that, turned one of them from incompletion to 7 points for the other team (nevermind that the officials got the call wrong).
When you add this to everything else about Ginn that fans don't like it's easy to see why he's been blamed the way he has been for those two losses. If it was Cam or Hartline or Bess that missed those TD catches in the Colts game, fans wouldn't be on their cases nearly as much.