Pick the route that is sustainable long term, mix the two concepts
I would argue that the GMail approach is more sustainable. Their 'invite a friend' tool and the fact that beta-trials tend to be quite 'exclusive' gives momentum to their signup ... I remember itching for a GoogleWave beta account for ages!
Overall, you have to look at your core competencies. Would scaling be a challenge? Where will things go wrong? controlled and sustainable growth in uptake during beta will give you space to tackle any problems but produce a constant growth, through your beta community until a non-beta launch and later critical mass.
A combination of the two can work well, think a "we are out of beta" party. Use your beta users as advocates to ensure word-of-mouth signups. You aren't Google, so getting the "exclusivity" part of a beta signup will be difficult; An alternative is to make beta users ambassadors for your site and allow them to recruit other beta users where space is (artificially?) limited. Think:
"Hi, Dave has invited you to be a beta user for http://somesite.com. We don't allow many people to test, but Alan thinks you can help us out!"
Give beta users additional tools, their own forum and other perks that normal users will not get. Above all, beta users can give you alot of value through testing now and advocacy later. Treat them right and you may have a small army of fanatics to help keep the signup rate rising.