A banjo “settles in” over time with all of the parts put together and kept at a certain tension, etc. Changing the head moves and resets a lot of parts of the banjo aside from just the head, that can each have their own impact on the banjo tone. Sometimes, this can mean a considerable difference in tone just from changing out the head. Also, many times the wear spots are not necessarily dirt as much as they are places where frosting has worn off which can also affect the tone.
I spoke with Russ Carson at Ben’s October Cabin Camp about a prewar Gibson banjo that I had seen him play in a video. I had also heard this banjo in other recordings before. I asked if it was as good as it seemed and he said, “It was…then he changed the head.” He told me that it had been great and it was getting back to that same greatness. But, I gathered that it had lost a lot of tone temporarily, and was taking some time to settle back in.
Now, aside from all of that, I personally like the looks of a worn head. Like @Mark_Rocka said, it’s kind of like a badge of honor and gives the banjo some character.