TF Accuracy sans 1 class

In an image segmentation task the background can be up to 99% of the whole image, and as a result the accuracy metrics really suck when training because the model goes to 99% quickly. Is there a way to train (or just output) without this included in the metric?