"I am cash offering on a condo w/HOA litigation, who is responsible for paying the HOA doc/transfer fee if I back out?"
I most agree with Brian on this one; meaning, you need to read your contract, specifically, Para 4D3 & 4D4.
"My agent mentioned that there are two costs buyers pay, HOA doc fee and HOA transfer fee (in this case, litigation report),.. "
I consider the HOA docs to almost always be a Seller disclosure (even with REOs), and hence, I predominantly have the Seller pay for these items. In fact, the CAR Standard Residential Purchase Agreement "defaults" to the Seller being responsible for requesting and supplying HOA disclosures using CAR Form HOA, which details what disclosures are statutorily required in addition to called for in the RPA. See Para 7A and 7B.
"....(in this case, litigation report), but only when the deal is closed, right? My deposit would be fully refunded if the deal doesn't get through even the HOA doc are ordered and reviewed, right?"
Not necessarily! The deal does not have to close in order for Escrow to charge for the documents provided. There is a chance the Escrow Company may charge a cancellation fee. If you received a Preliminary Title Report towards the end there's probably a section that states:
THIS REPORT IS SUBJECT TO A MINIMUM CANCELLATION CHARGE OF $400.00 AS REQUIRED BY SECTION 12404 OF INSURANCE CODE AND RULE 2 OF BULLETIN NO. NS-35E.
Can Escrow really do this?
Yes. The State of CA's Department of Insurance Bulletin No. NS-35E (dates back to 1965-66) allows the Escrow company to charge a fee - restrictions do apply. See: http://docs.Steven-Anthony.com/NS-35E_EscrowCancellationFee.pdf