Is it possible to serve models that have a distributed architecture, with multiple shards using tfx serving?

I’m planning to build a model that uses ParameterServerStrategy to distribute its parameters across multiple VMs with an assumption that it cannot fit into any one of the VMs. Is it possible to use TensorFlow serving to distribute the model across multiple VMs?

I was reading through this ( and I discovered that you need multiple servables for composite models that have multiple parts to them. But I couldn’t find any documentation that talks specifically about creating a cluster with tf serving, that uses multiple servables with each servable in a single VM.

Is it possible to use TensorFlow serving for very large models, where the models have its parameters spread across multiple VMs? If yes, can you please tell me how it can be done?

Thanks in advance!

submitted by /u/deathconqueror
[visit reddit] [comments]

Leave a Reply

Your email address will not be published. Required fields are marked *