First, I think it is a little more stressful to be a pitcher than a hitter on average. Secondly, there are several reasons why there aren't more 4 HR games (besides the fact that it is just hard to do) and they include:
1. The really good power hitters are usually walked once a game or so.
2. Hitters that already have 2-3 HRs in a game are more likely to get walked.
While these factors make it harder to hit 4 HRs because the pitcher takes it out of the hitters hand, it could be argued that there would be a lot more of these games if it was just talent and a luck based like perfect games for pitchers. For pitchers, they are in control of the game (for the most part, forget about managers for the sake of the argument) and get to call the shots. So, the perfect games are pretty much a result of just talent and luck unlike the 4 HR games.