There is no definitive formula for that.
IMO, the best way to start with reviewing the (closest) competition offerings, options and rates. Then objectively compare your site to those you reviewed for size, pageviews (inventory), how targeted your visitor base is, etc. There are some more ideas at http://www.powerhomebiz.com/122007/sell-advertising-website.htm
The rates very big time depending on the factors linked above but targeting has the greatest effect. Loosely targeted ads, legal service banners on a florist community site /extreme case :)/, may go for a low as $.5-1 CPM. While highly targeted - promoting pricey physician services/equipment on a physician only job site - may be happy to pay you $50-100 CPC.
Specific placement rate also vary based on the banner size and positioning (above or below the fold)
Although ad rates went down in the last couple of years, I keep list rates on a software developer community site at $3-$9 CPM but rarely charge as much - work out lower rates with each advertiser individually.
A CPM (cost per 1000 impressions) pricing is better for the start since you won't know how well the ads perform until you gather initial statistics. Once you have the click-through data, you'd know if you want to offer a CPC (cost per click) and what rate would make it comparable to the CPM charge.
Depending on you relationship with this first client you may be in position to ask them what is the avg rate with other sites and what they'd be willing to pay for the trial with you - a great way to start.
There will be advertisers that will low ball you and/or ask for a discount - you should structure your rates for that. You should also structure rates to be able to offer long term discounts. I have had few advertisers who would go with the list rates for a couple of month trial and then get back to me with the results and let me know what rates work for them in terms of ROI, if the rate is mutually acceptable I would have them for at least a year or two.
Building relationships with the vendors that advertise with you is just as important as if you were selling a product - good mutual relationships build long term advertising contracts. Be proactive - make sure to review the effectiveness of the long term campaign (every few months) and suggest improvements (bonus impressions, additional placement options) to optimize their click-through performance. They will love that and stay with you longer!
Whatever you decide as far as the banner options, for your own sake never (like in ever) do the nasty things like pop ups/pop unders, over the page, road block, ContextLink/In-Text - these will annoy your visitors. Stay with the less obtrusive options - your audience will appreciate that.
Good luck and let us know how this works out for you!