You either do nothing and just wait for them all to settle down - they will in time. Or you can remove a couple of the bossiest hens and put them somewhere else for a week, thus changing the pecking order. It can always be a bit tricky introducing a lone hen, sometimes they cope ok and sometimes they don't.
You may get similar issues with the new batch of chicks, or you may not, but putting several newbies in together is usually easier.