Yes, lack of colostrum would be the cause - there's no substitute for mums milk.
I haven't had personal experience, but know breeders who have bottle reared orphans and they just never seem to do as well as others who are with mum. I know of a few who have died suddenly for no obvious reason as young adults.
Obviously pregnancy and farrowing puts a greater strain on all their vital organs and reserves, breeding from anything other than a 100% healthy pig could be a recipe for disaster and heartache. Ultimately it'll be each person's own choice based on the stock in front of them, but I wouldn't risk it.