Customize TF object detection API

Let say I have an object detector from TF object detection API. Not much of customization can be done to the detector itself, but I want to add a branch on the detected images with LSTM to generate a description. Is there a way to do it in a single architecture? Or the only way is to train the detector and then train seperately the LSTM on detected images (2 stage)?