Layers have additional calcium in which is not great for young birds. Growers is lower in protein than layers - I checked the other day - but I'm not sure what effect that would have. I'd have thought the main problem would be the lower calcium for your laying hens so you'd have to provide additional soluble grit for them.
My first hatched chicks and ducks of the year (now 10 and 9 weeks respectively) are up in the main run but in separate pens from the adult birds. Normally the ducks wouldn't be a problem but one of the adult drakes clearly sees the younger ones as a threat and one time managed to break through and try and destroy one of the younger ones (and they're really not that much smaller). On the other hand, the cockerel in the pen next to the chicks has shown no interest in them. I wouldn't want to risk it with four week olds though - my second phase of chicks hatched 17th April and they're not going up yet awhiles - not least because I think they're still small enough a crow would have them.