You're either having fun with this or you truly need to sit down with a local expert and have a complete consultation.
Looking to buy a house between $1M and $5M is like looking to buy a car that would cost between $10,000 and $50,000.
Can you think of a few ways a $10,000 car may be different to a $50,000 car?
Similarly, there are likely to be great differences between a $1M house and a $5M house.
In the end, once you sat and drove the $50,000 car, do you believe (assuming money is no objection), you will end up with the $10,000 car or is it possible you will be lured by the added luxury of the $50,000 car? Perhaps. It will all depend on a number of factors ranging to your perception of owning a used car, a type of brand and whether you need to impress yourself or anyone else while driving it.
Similarly, after you've walked through a $1M house and then a $5M house, do you think there may be some perceived and real differences perhaps to include the area, the size, the amenities, its features, it's quality, and again, even the way you feel as you see yourself living in one vs the other and the impression that makes in your and your friend's psyche? Will you and/or they be impressed more with one vs the other?
Again, that will also depend on many factors, including the cost to keep up with one vs the other, and whether your discretionary funds used to purchase one vs the other is more or less depleted.
What else would you do with $4M or will you just want the $5M house after all?
These and other questions should be explored during your consultation. Don't be shy. True experts like myself will gladly sit with you and help you sort it all out.
Who knows, in the end, you may end up really indulging your well deserved self - guilt-free.