Training not working on M1 mac

I am training a VGG16 net with cifar10 dataset on my M1 mac, and I get this error shortly after training starts. I have tensorflow metal, and training works for smaller datasets like MNIST, so what is going on here?

Error: command buffer exited with error status.

The Metal Performance Shaders operations encoded on it may not have completed.

Error:

(null)

Ignored (for causing prior/excessive GPU errors) (00000004:kIOGPUCommandBufferCallbackErrorSubmissionsIgnored)

<AGXG13GFamilyCommandBuffer: 0x1631088b0>

label = <none>

device = <AGXG13GDevice: 0x11f98be00>

name = Apple M1

commandQueue = <AGXG13GFamilyCommandQueue: 0x1613a4800>

label = <none>

device = <AGXG13GDevice: 0x11f98be00>

name = Apple M1

retainedReferences = 1