If your fence is moldy and an eyesore, then I would recommend replacing or removing it. Look at your house and property as a whole picture. If one or more things are not satisfactory, then it is time to judge what needs to be done. I am very good at staging properties and getting them sold quickly because a well staged home brings forth emotions in buyers and gets your property sold. I would be happy to help you make decisions on what is really important to do.
I would reinstall the front yard. What a terrible disappointment you had after all the work you did. I would recommend putting in a few small trees for the proper look in a front yard. You have time for a yard to get established since you aren't moving until 3 - 4 years.
Thank you for the question. With out seeing the house or the comparables any work you put in the house may not come back to you when you sell. At this point since you are not ready to sell only do the work for things that you want to enjoy. When the time comes invite us realtors in for an opinion as to what other work could help you sell for more or sooner.