When to remove trees or fruit bushes: the easy answer is when they are dead. Care needs to be taken with the dead plants since they are usually hosting a disease of some sort. Remove and burn the plant or compost it far from your orchard. Remember to clean and disinfect all of your tools afterwards.
For low performing or weak plants, one needs to weigh the plus/minus analysis as to why the plant should stay. If the fruit is wonderful, I will prune, adjust the nutrition and coddle a plant for a few years to see if the vigor returns. After that period, for the health of the whole orchard, I remove any plants that could spread disease to the other trees or have your reaching for the strongest chemicals.
If possible, try to identify the disease that weakened your plant. There are varieties that have greater resistance to specific problems and perhaps you should consider one of those when picking a new tree.
If the plant is healthy but the flavor and quality of the fruit is low, I will remove it as well. There are too many wonderful varieties available to settle for mediocre fruit.