Your question is the primary reason most people don't go to a startup.
If you are comfortable and safe where you are, why risk it, is usually how people deal with talking themselves out of the risk and potential failure.
I have been at 5 start-ups. Two of which failed, two are still around and one got bought. The emotions I went through on the first failure were pretty awful. Most of the feelings centered around my disappoint in my own performance and how others would see me as a failure.
It turned out, I was way too hard on myself.
Those emotions turned out to be self-centered. I was naive as to how others would look at me. It turns out, no one really cared. My real friends were supportive and helped me find another gig. Sure, people asked why it failed and some acquaintances thought I was a loser but who cares. The people that mattered most just wanted to help and see me get back in it.
You realize that most people want to take the risk but are too afraid to try. That's why they might give you a hard time about failure. In reality, they envy you for trying. They rationalize like you did about the "stable and safe job." In reality, none of us are safe in our jobs.
I have countless friends who have been at their "stable" jobs and got laid off. Poof, 20 years at the company gone. The sad thing is, they got comfortable and now finding something new is a lot harder.
Your real friends and colleagues will not think negativity on you if your startup fails. The world is full of failed startups. Every successful entrepreneur has had a flame out -- it's part of the rights of passage.
The potential failure should not discourage you from giving it a shot. It's not how many times we fail -- it how many times we get up and try again that matters.
Don't get me wrong. Shutting down a company is painful but not the end of the world. You learn, you grow and you move on.