I will presume you are in England, sorry if you are not, others will no doubt know the detailed rules for Scotland, Wales or NI.
The following is my current understanding, and I will repeat at the end, but you do need to chat this through with the RPA to ensure it is correct.
You can get a holding no. on any area of land that you have use of. As far as I know, you do not need a formal tenancy agreement etc. to get this, as the CPH is merely a way of identifying a piece of land, and your name make you the responsible party for compliance with some (if not all) legislation affecting that CPH.
Land with a CPH can be spilt, or amalgamated (within certain parameters) as needs arise and by informing the RPA. You can also “own” multiple CPH’s, as the CPH applies to the land, not to you, and a single CPH can apply to multiple parcels of land within 5 miles of each other for sheep and 10 miles for cattle.
If this is not a new farm, it almost certainly has an existing CPH against it, and presumably you are not taking over all the land, just a part. If so...
In your case, if the farmer wants to keep his rights to move animals on and off his bit of the farm, one suggestion would be for you to get a separate CPH for your part. This will then mean that you are independent of each other for livestock movements, and if he decides at some future date to have animals, then he can go ahead.
However if you do this you must be able to move animals on and off your holding without them setting foot on his holding (and visa versa). Eg you cannot unload in his yard and walk them down to your land, as the unloading will create a move to his land, and then a standstill will prevent them moving off his to yours for 20 days !! As far as I know (and RPA should confirm) you can drive through his land to get to yours, but any loading or unloading must be done on your holding. So you may want to look at how you would achieve this if this is the middle of a muddy field in winter, and you need to get a pig in or out.
Alternatively you could check that you could do this at a later date if he goes back into livestock, and get the current CPH transferred to you, so that you are responsible as you don’t need to split it immediately. I suspect that he would be less happy with this, but if he has no livestock, then you could then unload in his yard, as you have the entire CPH.
Finally you will both need to ensure that this does not affect his Single Payment Scheme. I would suggest to keep him happy, that you do a written rental agreement - doesn’t need to be complex, just state what you are renting, for how long, and what payments you are making, together with any responsibilities eg who pays for water used, hedge cutting, maintaining fencing and land restoration at the end (not a full list). This could then also state that the land you rent is his on the appropriate deadlines (for 2011 may 16th in England, and May 15th for Scotland, Wales and NI) - in other words that you do not rent it on those dates – to Claim SPS the land must be at the Claimants “disposal” (ie the farmer if he is to receive payment) for the whole of the deadline day each year. Again SPS is horrendously complex (it takes 115 pages to describe how the scheme works, and around the same again to explain what you need to do to comply!), and you will want to check that you do not affect this payment. If he doesn’t claim SPS, then heave a big sigh!
I will re-state that all this is horribly complicated to set down as general principles, but take you proposal to the RPA and tell them what you plan to do.