First of all ask yourself if you can and want to run cost per action advertising or simply display advertising.
Think of CPA like an affiliate program where you get paid when a user goes to another site and purchases a product. The other option is a display model where you basically get paid based on how many people see the ad, so having high traffic here is important.
I would start off with some market research:
- What do competitors charge? If you have no direct competitors I would find another niche site that pulls the same amount of traffic in the same vertical and see what they charge.
- What does a similar ad cost that is placed through a major ad/affiliate network?
- Find out about your users. Age, income, etc.
- Find out about how advertisers in general feel about your user segments. For example luxury brands will pay more for ad placement when they know your users all make a certain amount of money.
Now if you do display ads you got to think about a pricing model. You can do "sponsorship" where someone pays you monthly to have their ads placed on the site. Another model is a CPM (cost per impression) where you get paid based on every 1,000 times that ad is seen. Another twist to this is you can enter agreements where you get a certain CPM rate, but also get additional payment if the user actually clicks the banner and goes to another site.
Based on what others are doing and general market rates you should be able to come up with some base pricing. But remember there will always be some trial an error until you find the right price mix.
Now future a price increase is going to depend on your site as well as how many people are lining up to buy your ads. If you start seeing 3-fold increase in traffic then you probably want to increase the ad rates. Same goes if you have more people wanting to buy ads than inventory. There is no hard and fast rule for this, so again train and error on how much to increase and when.
Finally, if you want to get into direct ad sales first build yourself a rate card. Think of it as a 1-2 page PDF brochure. The rate card should have an overview of your site, traffic numbers, demographics of your users, a brief why someone should advertise, and pricing on your various advertising options.
In order to start selling direct find potential companies you would want to advertise on your site. Write up a sample sales pitch and then leverage your network to see if you know anyone that can get you in contact with that company's advertising manager. Having a phone call or meeting with someone via a personal contact will be a huge plus over a random email sales pitch. Now this is going to be a long and time consuming process in th beginning that will take huge amounts of time for possibly little reward, but at least you will learn. Once you kind of have your market down, you could always hire a sales person to develop leads and make calls for you and offer them a high commission.
GOOD LUCK!