First, have your agent show you in the RPA, the language that states that a per diem charge was agreed to between the seller and buyer. if it was agreed, and YOUR actions or the actions of someone on your side of the transaction (lender, appraiser, inspector, etc.) delayed the closing, then you should pay.
If it is not in the agreement, or it was the seller's delay, then you aren't required to pay any charges. In fact, the seller might owe you, depending on what the RPA says.
The bottom line, read the RPA and understand what you signed.
*this is my logical & completely sane opinion, but you should consult an attorney as well*