Three is the magic number. You want it to be small, since you want the fewest obstacles to making a decision, and you want it to be an odd number so that there are no ties. Right now, you'll want the board members to be "yes men" who will vote the way you ask them to. This makes meetings shorter and everything easier as you are starting out. Later, if you get funding, your VC will take one of the board seats, you'll have one, and your VC will want to approve the third seat (usually some third party that you both respect).
Remember that your board's purpose is to make decisions, not to give you advice. You should create a separate advisory board with as many people as you want on it, but they won't sit on your official board. Often, you will ask your advisory board members for their thoughts on an issue and implement their advice without ever convening an official board meeting. But when you do need a board meeting (say, to grant equity to an early employee), you'll want the official board to "rubber stamp" your decision.