Within reason I don't buy on price. (See my strapline...) Other things can be more important.
I prefer to buy from a company with whom I have had good dealings before. If I know from personal experience that they are helpful and good at resolving any problems, that will go a long way.
In the absence of personal experience then things like recommendations on this forum, eBay's buyers ratings (to an extent...), etc, will help.
I'd buy from someone more local to me in preference to someone a long way away. A small / family business in preference to a national chain / faceless internet business. A company with clear ethics in preference to just some other vendor.
I am unlikely to buy from a company that doesn't publish a realistic looking address, ie, an address of an actual place not a Post Box or handling facility. I probably won't buy from a non-UK company unless I have very good reason to feel comfortable - not jingoism but feeling protected if something goes wrong! (I don't know the legal facts but inherently feel that there would be legislation to back me up if I buy from a UK company and I am not sure how much protection there would be for buying from a non-UK company.)
I probably won't deal with a company that is likely to bombard me with offers, etc, and like to use ones that make it clear I would need to opt in to any mailings, etc.
I won't buy (unless there is no other suitable vendor and I really have to have whatever it is) if PayPal is the only option for payment. I do have a PayPal account but I object to being forced to use an offshore financial instrument and thus be unable to have any protection from our Financial Services legislation. (I won't give PayPal access to my bank account, I always use my credit card if I do purchase through PayPal, then at least I have fraud protection through the credit card.)
There're probably quite a few other specifics, but hopefully that will give you the drift.