Oh Luce no, this worries me too. Its a tricky issue

Children just open their mouths and words come out without them thinking but they also say what adults are too scared to. That's not to say your house is dirty!! Just that children are honest and harsh critics.
There might be an animal odour there that you are all immune to having lived in the house but when a new person comes (who quite possibly lives in a sterile, bleached, animal free environment as too many kids do these days!!) then they might pick up on it.
I remember when I was a district nurse, there was one house we had to visit and it was AWFUL. Cluttered, full of cobwebs (they were so bad it was in your hair and everything) mouldy pot plants, mouldy coffee cups - you name it. I think it was a health risk in itself but we used to have to go there and do the lady's dressings, made us want to baulk. - now I don't reckon your house is like that is it?!
If you are really worried, do you have a really good friend who you could ask?
Our house isn't perfect, my husband has OCD so he drives me to distraction with constant tidying (and I mean totally to distraction, every waking minute) but I make sure the kitchen and bathroom is clean as a priority then vaccuum as and when it needs it. Dusting (yuk) little as possible. Like you, my children are clean, uniforms and clothes ironed and their hair is tidy.
Try not to let the comments on the little ones get to you. Once one says it another will jump on the silly bandwagon - its probably just they've never smelt small chicks before, how lucky they are to have your son as a friend as they can actually come and visit all these animals!!
Lisa x