Is the limitation on the bedrooms due to septic? zoning? And what is their policy on rebuilding what currently exists? These are the questions that need to be asked to the building department. Only they have the authority to grant the permit to replace, rebuild. No REALTOR is authorized to make representation regarding the property that conflicts with the town.
If you have a issue that needs mitigation with the town, contact an attorney. It is well worth the investment.
Only the town has the authority to issue permits. An agent or appraiser's statement may be taken into consideration, but in the end, the town will decide, will issue permits, certificate of occupancy, etc.