The worst part of working at Google, for many people, is that they're overqualified for their job. Google has a very high hiring bar due to the strength of the brand name, the pay & perks, and the very positive work culture. As a result, they have their pick of bright candidates, even for the most low-level roles.
There are students from top 10 colleges who are providing tech support for Google's ads products, or manually taking down flagged content from YouTube, or writing basic code to A|B test the color of a button on a site.
Some of the downsides of so many overqualified people:
It's hard to get promoted quickly, since the person above you as well as at your level both have great educations and strong work ethics. When it's standard to be awesome, and the work isn't particularly tough to begin with, it's hard to differentiate.
The work may not be intellectually rewarding (read: boring). It can be tough to feel a sense of accomplishment about what you do, and that sense is actually quite important to the type of people who are ambitious enough to get over the Google hiring bar.
Some people end up losing their drive by working at Google. They get accustomed to not trying their hardest, but still having an awesome day-to-day life.
Some caveats: Many Googlers are clearly among the brightest in the world in their field, and they're able to run full stride in their work. If you take your career into your own hands, you can find a role that challenges and stretches you as much as any other job in the world.