@patrickrhone This is a great idea. A few thoughts:
Location would be really important, especially regarding urban vs. suburban, type of neighborhood, streetfront or not, managing takeout traffic in peak hours, and proximity to pre-existing libraries and bookstores.
I like the quiet idea, but you would need to make some calls on silence or what type of background music. I already frequent a music-less coffee place in Union Station in Toronto that feels like an oasis in a sea of people. It serves pourover with freshly ground beans (also the option of pourover or tea), but I think the burr grinder is dampened so it isn't that loud. They have a small book collection, but also specialize in Italian baked stuff. No seats in the actual store, with a few standup tables outside. The staff are trained to foster conversation with customers, which is how they develop bonds and get repeat customers. The only thing I don't like about this place is that I can't spend any time there sitting and reading.
There are some reading-focused cafes in Toronto that are worth a look.