You need to compare your lower monthly payments with up front points (cost) to slightly higher payments with no points to determine where the "break even" point is (your mortgage broker should be able to help you figure this out). If the "break even point is, for example 60 months then you have to decide what are the odds that you will own the home/be paying the mortgage for more than that point (5 years in example). If you plan on living in the home with the same mortgage beyond this point, then buying the points is worthwhile. Otherwise, it probably is not.
The answer really depends on how long you intend on keeping the property. For many people having a lower is the ultimate goal when purchasing a home. The cost of buying down the points may not be worth it if you do not intend to keep the property. As an example, if it costs you 6,000 to buy down your rate and it saves you 100 per month, then it would take you 60 months to make your money back. If you only plan on keeping the home for 36 months, it would possibly be a loss. Evaluate your long term goals and speak to your loan officer as to the best way to proceed.