I most certainly wouldn't take him back.
You said you had him confined in a trailer so they had every chance to examine him. He is still suitable for breeding which is what he was bought for - it would have been another matter if, say, he'd only got one bollock. It's not like he's got a large white patch that you tried to conceal, and I think that a few white hairs would probably not be passed on anyway.
But most important of all, as mentioned, is the disease risk. This is also the time of year for transmission of scab. I once lent out a ram. Unknown to me the people who borrowed him bought some more sheep from market and these had scab. They brought my ram back and never mentioned it and it went right through my flock. It took a while to establish and finally became evident just as we started lambing.
This is a classic case of caveat emptor and there would seem to be no valid reason why you should take the animal back.