Specialty bookstores are a bit of a mystery to me.
Why limit your scope? Trust me, you're going to want as many customers as possible. There is no reason not to cater to as broad a scope of people as possible.
Here's the thing. You can still focus on one element of the store without dismissing the rest.
I arrived at this conclusion through trial and error. Basically, the beginning core product of Pegasus Books--comics--could never generate enough business to pay the bills. So I was always forced to diversify. Pop culture toys and accoutrements, graphic novels, board games, card games, and so on. Not to mention products that we carried when they were hot--beanie babies, pogs, non-sports cards, etc.
The trick was to add product lines without detracting from the existing product lines. It has always been music to my ears to hear customers of each category describe my store as that--comic people saying I'm a comic store, book people saying I'm a book store, etc. That means I've succeeded in broadening my scope without turning away the other possibilities.
So to become a Science-fiction store, or Mystery store, or Horror store, a Children's store, the question becomes--are you going to attract more of the specific customer to cover the loss of every other possible customer!?
You don't have to go crazy. I've carved out every niche of fiction I can, even if it's just a shelf or two. I originally was going to skip non-fiction, but I manage to spring loose three bookcases and fill them with the best non-fiction I can find. It's enough. I've done this will a little over 1000 sq. ft., which is a modest size by any measure.
I cater to readers. That is, I carry enough diversity that a reader is likely to find something they might like. I may not have a specific book they want--but I learned long ago that the chances of having an odd book are near zero, and would be even if I had ten times the inventory. (People seem to have no real concept of how many fricking books there are in the world!)
But as long as I have a nice selection, there's a decent chance I can hook a reader. I see no reason not to have a nice selection in as many categories as I can do.
My door is open to everyone.