How would I determine my start-up's target market share and translate that into financial forecast on a commission based business model?
Step 1 - Determining Size of the Market
You could get access to analyst reports that cover your industry or a particular company such as Groupon. The more meaty analyst reports provide "size of the market" and other industry specific information. If you have access to a brokerage account you can probably get one for free or inexpensively otherwise the reports can be very expensive.
If you are not able to get the analyst report then check out Groupon's or other similar publicly traded deal site companies SEC filings and they may have info to give you an idea on size of the market.
Step 2 - Market Share Capture
You will need to build a financial model based on the key drivers of your business model. For example, your superior marketing efforts give your business a 10% conversion rate for email addresses in its database with an average spend of $10 per client conversion and then you would calculate your commission off of that.
Hope this helps.