Ordering is both a matter of math and of intuition.
The math accounts for most of your orders: I check the inventory of new titles and if the wholesaler has enough faith in a title to carry a large backstock, I am more likely to order. Every week I look over the coming titles and try to order what I consider to be the most important.
For us, that accounts for about 20% of our orders, while the other 80% is restock or books that appeal to me. I'm fairly certain this equation can be reversed for most indie bookstores--that is, most of the weekly arrivals in most stores will be the newest new book, and probably only 20% are midlist. I won't go into all the reasons for this again, but it seems to be our reality.
The intuition part is more fun. It's a matter of, "How clever can I be? What mix of books will both appeal to people and also make our store unique?"
Every store is probably tailored to the owner's interests, at least those still in charge of humans and not algorithms. We all have our biases and our favorites. The test, in a way, is to see if what your favorite is popular enough to sell to others.
Way back in high school speech classes, I listened to a senior expound on the idea that if you like something there's a good chance that others will like it too. That seems to be generally true. Not everyone, but if you aren't too far off the mainstream, enough to keep you going. Or more to the point--the average thing goes by without notice. but there's a good reason you like something enough to mark it.
So I trust my instincts.
One of the first things I did when first buying new books is start getting titles that I remember liking. I've read a lot of books in my life and I've settled on authors I like, books I adore, and books that intrigued me.
So that's a good start.
Then I started listening to the customers. I find that almost any book that a customer asks for is something that might be worth carrying. I have to be careful that it isn't something odd that only that one person likes, but you can usually tell. A little research will either confirm or raise doubts.
So I order that requested title and often it will sell to someone else as soon as it arrives and I have to order again in case the original customer comes back.
I also listen to my quirks. I can't tell the number of times I've ordered something that I was fairly certain only I was interested in only to have a delighted customer find it. That's cool.
I'm partial to art books, for instance. They don't sell fast, but I still like having them around. They make me feel good.
I guess the point of all this is to say that it's important to have product you can stand behind, even if it isn't the most popular thing around. That's what make it interesting.