I've been reading Leonard Bishop's Dare to Be a Great Writer, a 1988 compilaton of "329 keys to powerful fiction." While I don't agree with all of his advice, sometimes things are just spot on.
In a section called "Don't Postpone Your Novel," he lists all the arguments to do so. It takes years to write a good one. You don't have that kind of time. You won't get published anyway. No agents are interested in new writers. You're probably not all that good to begin with. Maybe you should take some more classes first or go to a few more workshops or find another writing group.
Instead, he admonishes: Ignore the practicalities and embrace the vision. Believe in yourself, your abilities, your talents, your sense of self. Hone your discipline, your persistence. It's that old saw about courage. That courage isn't about waiting until we have no fear, it's about moving forward anyway. And so what if your first novel isn't all that great? At least you'll have written it. That's more than the talk-about-writing folks can say.