I see two sides to this.
Would you enjoy doing it full time? Or is it better suited / more rewarding keeping it a hobby. If you start working on it full time, your hobby can turn into work. If you really love it, it could turn out to be a great proposition. To quote Confucius "Choose a job you love, and you will never have to work a day in your life".
The other one has to do with what you have to leave behind. Do you enjoy what you're doing right now full time, have opportunities to work on something you couldn't on your own, gain experience you couldn't on your own, or are you part of a great team? Do you want to leave that behind? What do you want to do? Do you want to grow professionally more? Would it be more useful to stay there longer?
But the most important by far is if it makes you happy.
If you're happy keeping it as a side project, it's fine. If you love your job that's fine too.
Really that's the driving factor. Follow your passion. Money is a motivator, but it's a poor one if it's not backed by passion.
There's no requirement that everybody has to start a company. Working for a company you like, doing things you enjoy are equally rewarding.
If you don't really like that job, or don't find it necessarily rewarding, ask yourself what's keeping you from jumping on the bandwagon.