This is a great question and one that can be more complicated than what you might expect.
The purpose of an appraisal contingency is to protect the buyer in the event that the bank does not appraise the property at a price that is at least equal to the sales price. If the property does not appraise, then the buyer has the option of canceling the agreement. In the real world, the buyer can and often will approach the seller about a price adjustment, which the seller is not obligated to accept. The buyer also has the option of bridging the shortfall with additional cash to make the deal work.
The loan contingency specifies that the contract is contingent upon the buyer being able to obtain the loan rate and terms as specified on the first page of the contract. It is not specifically tied to the appraisal. In your example, if the buyer is putting 20% down, but is still able to get the loan amount and terms specified even if the property did not appraise, they would be obligated to proceed (they might qualify for a 10% down loan at the same rate and terms). This is especially important in cases where the buyer is putting large amounts down (more than 35%).
Let's say the buyer waives the appraisal contingency. If the buyer, for example, is putting 50% down on a $1,000,000 purchase price, and the property only appraises for $900,000, the bank will probably still make the loan of $500,000 since they might only require 30% down, even though the house did not appraise at the sales price. The loan contingency is tied to the loan amount and the rate and terms, period. If the buyer is able to get the stated loan amount for the rate and terms specified, even if the property did not appraise, then the buyer is obligated to proceed. The fact that the property did not appraise is not a factor if there is no appraisal contingency. In this same example, if the buyer did have an appraisal contingency in the contract, they would not be obligated to proceed.
One other thing... before you remove your appraisal contingency, make sure the lender has completed an appraisal review if it is part of the bank's final approval. I have seen cases where buyers remove the appraisal contingency, only to find out the bank has reduced the appraised value as part of an appraisal review before they give final approval. As always, check with a good real estate attorney to discuss the details of your case specifically.
I hope that helps!