There are at least 3 RV places in town. One, Manzano's (575-538-0918) just outside town is in a beautiful location with great views, wildlife walking through, neat, clean and operated by a very nice couple. Depending on your finances this would be the best bet for the description you gave of what you're looking for.
You can camp free in the Gila National Forest, but not for long stretches at a time. Rangers keep you moving every so many weeks. Depending on the size of your rig there are some size limitations in certain areas of the Forest where large rv's would have trouble making the grades, etc.
There's a KOA too near Manzano's.
The Silver City RV park is right smack in town, and close to everything. 575-538-2239. This wouldn't be as quiet however as the ones out of town of course, but SC is not that noisy to begin with. Hope that helps.