What ad sizes and formats should we use, and what other factors should
we be thinking about before setting any prices? Also, what is
acceptable in terms of raising prices as our traffic increases?
If you are thinking about getting into the business of selling adverting, the place to start is the Internet Advertising Bureau (IAB) website. Certainly, you should definitely be using the IAB standard recommended sizes. This will save lots of trouble relative to having advertisers submitting you creatives. Also, there are very standard contracts used for working with advertisers and their agencies. See the terms and conditions section of the site.
Next, relative to pricing, my first recommendation is to start out as absolutely high as possible. This is for three reasons:
1) It's hard to know your prices are too low. Customers will tell you if they are too high and you can always negotiate a deal down to what they can afford or what they think the traffic is worth to them. At the beginning, every deal closed is realistically going to be a negotiation and if this is a very niche thing, it might stay that way forever.
2) From a customer perspective, it's always easier to lower prices than to raise.
3) As time goes on, presumably your inventory will be increasing and so there is a supply and demand thing going on. If you have limited high quality niche targeted inventory, you want to be selling it to the company that is willing to pay the most for it. Setting your price high then is selecting in such customers. After you have more inventory available, by implication you will probably want to start lowering your price to fill the inventory and in the end be optimizing total revenue generated by the site.
Relative to when you start to consider selling advertising, you have to have enough inventory available to even make it worth your trouble. For example, let's assuming with a decent niche targeted site you could get a $10 CPM. If you had 100,000 impressions to sell per month, this would be only $1000/month. Worth the trouble!? Don't know. You tell me. Now if this is really high quality niche stuff and you could get a $100 CPM or maybe even a $200 CPM, then that would be $20,000 month and probably well worth it. Not a lot of sites can get those kinds of rates, but they definitely exist for vertical niches.