But having done all these things I can say with authority that they were the most invigorating times of my life. Risk-taking (and in retrospect, they were small risks) stimulates the brain. Hard work - and reaping the rewards of it - is one of the most satisfying things your brain can do.
If you live a fulfilling life - are busy with your work or other pursuits, you don't have time to "hang out" every day with people. Sure, you may socialize on occasion - which makes those occasions all that much more special. But doing the same thing over and over again doesn't make it better or more interesting. And socializing is one of those things that is actually better in moderation.
If you find yourself constantly shouting down more successful people in life, ask yourself why. Maybe what is really bugging you isn't them, but your own life.