Cheap web hosting (~$5/mo) is basically a commodity these days, and there are dozens of services that will fit your needs. However, most of these companies keep their prices down by skimping on services.
When I first got started with web hosting, I signed up for a super cheap package with Just Host (don't use them) that worked fine for a few weeks. But when I had a simple request (wanted to change apache configuration so URLs wouldn't need ".php" extensions) it took them almost a week to respond, and then another few days to actually make the change.
If you're going to build a business around a website, you need much higher quality service than that. You can probably imagine how frustrating it would be to wait a week for a response if your server crashes or needs some kind of critical maintenance. Dismal service means lost customers and revenue.
As I said, the actual server hardware is basically a commodity. What you want to shop around for is the service that backs it all up. You want a number that you can call 24/7/365, in case something breaks in the middle of the night on Christmas. You want people who are geeky and highly knowledgeable about their equipment, so they know how to help.
For that, the only company I'd recommend is Rackspace. I've been blown away by their friendly, helpful support staff multiple times, and have moved all of my sites to their servers.