Training custom EfficientNet from scratch (greyscale)

I’m looking at reducing the costs of EfficientNet for a task
that only deals with greyscale data.

To do this, I need to reduce the number of filters across the
network by 1/3rd (RGB -> (B/W)), and train on COCO in
greyscale.

The
TensorFlow 2 Detection Model Zoo has
a link of training configs if you want to train from
scratch.

However, I can’t seem to find how I would edit the architecture
to reduce the number of channels.

I can see there’s an
official definition in Keras, however I’m unsure if this is
what’s used by the config.

If there was some way to load the saved model, and then edit
it’s structure that way, that could work. But I’m unsure if there’s
a better way to do this.