If you're planning to "age in place," then this sounds like a great idea and it should make the home much more convenient for your use over the long run no matter what your age. If you're planning a move in the next few years, you'll want to consider how the change impacts the floor plan carefully. My buyers have been less and less interested in formal rooms in recent years, but make sure your remaining public space is big enough to not feel cramped with the removal of those rooms. Best of luck!
If you have a good sized den and a good sized kitchen, this plan of yours may not really effect the market value of your home, but I have always suggested to clients before doing such a major change to the floor plan of their home, consult with a couple of local Realtors and see what they would suggest. It really depends often where your house is and what is the age group of buyers looking in your neighborhood - what do other houses have in the neighborhood.
You cannot do things for the next owners; you can't make everyone happy and you'll drive yourself crazy.
This is supposed to be one of the primary benefits of owning a home; having it fixed up the way YOU want it.
Having said that, it would have to be seen and visualized for a good opinion:
If you would still have a big Family Room; you won't miss the living room.
Do what makes you happy!