Sounds like you talked to the wrong insurance agent.
Ask your Realtor for some recommendations.
I don't know specifically about your area, but it's true that there can be some horror stories. The worst situation--and this goes house by house--is when a house is uninsurable because of the number of claims by the previous owner. So you could have two side-by-side identical homes. One owner has made repeated claims and the other hasn't. It could turn out that the one house is uninsurable, while the other one qualifies at a nice, low rate.
There are also issues dealing with flood plains, coastal areas, etc. That would affect all the nearby houses.
Finally, of course, there are variations based on all sorts of factor pertaining to the house--size, proximity to a fire station and a fire hydraunt, type of construction, age, etc. But you already know that. Those, though, shouldn't produce horror stories, just some variation in rates.
So, again, ask your Realtor to recommend an insurance agent who can help you. That should do it. If it doesn't, you can always ask to see the seller's insurance policy. It might not be quite the coverage you'd be looking for (the deductibles might be different, for instance), but it would at least give you a ballpark idea.
Hope that helps.