Honestly, you need to be speaking with a lender as they are the ones loaning moeny. The agents are right, much harder to buy land or build a new build for many reasons. Simply put, the lenders don't want to lose more money on a new build when there is so much inventory available at the current pricing. I strongly urge you to speak with a lender first before you buy any land. I had a client who had the same thoughts, bought the land and then after ward was upset when he couldnt get a loan to build on it. I honestly don't like being "right" in those situations when it means my clients lose money.
Most conveintional loans will require 20% on land, but my point is you need to be speaking with a lender about that, not agents. Best of luck.