This is a good question -- it's smart that you're looking to selling your software on your own website. Now you get to keep a bigger cut of the app price.
Keith's answer is right about the steps if you want to build it yourself. Or a more broad overview of what you need is this:
- A payment provider: It doesn't need to be a merchant account, but rather it can be anything like FastSpring, PayPal, or any one of the thousands of payment providers. There are some big hurdles for getting a merchant account (rather than something like FastSpring). One downside to a merchant account is that you need to maintain PCI compliance. This can be expensive.
- A licensing system: Since you'll be operating outside of the Mac app store you'll need to add your own licensing to your software. You can either build this yourself or purchase from a third party.
I'm founder of the company that makes LimeLM -- a licensing solution for Mac, Windows, and Linux. We have articles and some "drop in" code written for Authorize.NET, PayPal, MoneyBookers, and FastSpring. You can just take the code, modify the price and a few other configuration options, then upload it to your website.
Our customers prefer the FastSpring option (see: Automate license generation with FastSpring) because they have to add 0 code to their own website. You can just paste the license generation code directly into your FastSpring dashboard and add a link to the checkout page on your website. This way a customer can just checkout through FastSpring and get their product key emailed to them immediately.
We have a free plan if you just want to kick the proverbial tires before you commit. Also, FastSpring has a "sandbox" account that you can try out as well. So, if you're looking for a ready-made solution then FastSpring+LimeLM is a good place to start.
Tell me if this helps.