In the olden days, before the internet, you had to go to the dreaded dealer or pay the fees of an auctioneer.
Both these gentlemen (I use the term in its most all-encompassing form), would look sadly at your offerings and you would leave (much later if you sent them to auction) feeling disappointed with the cheque.
The internet has changed all this. Now you can hunt for the book you have on several sites which provide an online bookselling service. This gives you an idea of what it is actually worth, which is a good starting point.
Amazon is where I start. Search for the book and check the price of the cheapest copy in similar condition to yours. Now you have a decision to make. How much is your time worth? I don't bother to list a book for sale on Amazon unless the price is £5 for a hardback and £3 for a paperback. Be careful if you decide to go for a lower price as the Amazon fixed postage allowance may not be enough to cover your costs. Then, once you have opened a seller account, you just have to enter a description of the condition of the book (definitions of the grades are here ), enter your price and let the listing run
Amazon take a high level of commission and fix the postage charge but, on the up side, there are no listing fees and you always get paid.
Setting the price is quite straightforward but if another edition is available much more cheaply, you are unlikely to sell yours (unless it is a collectable edition), so look at the prices of other editions before pricing yours.
And then there is the problem of optimistic, usually US, booksellers. Some books are available at totally ridiculous prices. These books are not going to sell and nor will yours even if you price it at £2 less! To detect genuinely expensive books search Alibris and Abebooks. If the price is high on those sites, then you probably have a real rarity. You can sell on these two but they charge a monthly fee and it is only worthwhile if you have a very large number of books to sell. I don't know much about selling on these sites though buying on either has always been a good experience.
For really rare books, say over £50, condition is so important that I don't try to sell them online. I wouldn't spend £50 on a secondhand book without seeing it in Real Life so I don't expect anyone else to do the same. Those I send to a specialist book auction.
One last hint - Keep your inventory organised or you will sell books and not be able to find them!
I can't say that I've found many venues apart from Amazon very useful for selling books.
If you try them on the obvious auction site, do check that there are not dozens of the same title on already. Check completed items of the same title or at least the same sort before listing yours. If they go through unsold, don't bother to list yours.. move on to my next post!