Adjust your thermostat - The U.S. Department of Energy estimates that by adjusting your thermostat back 10 or 15 degrees for eight hours, you can save 10% a year on heating and cooling bills!
Installing double-paned windows - Homes with old windows can be add up to an extra 25% of your heating bill. The Department of Energy has a list of models and brands of double-paned windows that have been given its Energy Star brand of approval.
Weatherize your home - Apply weatherstripping or caulk to drafty areas and cracks to keep the heat inside and the money in your pockets.
Use a space heater - If youâ€™re only looking to heat one room, a space heater, which runs on electricity, can deliver warmth at a fraction of the cost of oil or gas.
If the back up system is electric, you would have 100% efficiency because all of the energy used to create heat would come into the house, but the cost of back up electric heat is 2-3 times that of a 90+ efficient gas furnace.
Gas as an energy source is less expensive, but some heat goes up the chimney. Gas furnaces are less expensive to purchase, probably by half or more of a comparable heat pump too, but a heat pump provides summer cooling.
So did you want the most efficient or least expensive?
A short answer to your question would be smart design including placement on the lot, passive solar, attention to the building envelope and much more. As a retrofit I believe an electric heat pump in the way to go. That way you will be ahead of the game when renewable energy is more common. They have been in Asia and Europe for a long time. If you want to get more ideas I suggest you check out the NW Green Home Tour on April 27th in the Seattle Area. It's a free tour where you will get to see these concepts in person. Just look it up online.