Tensorflow GPU failing with different error flavors but CPU works

So I have been trying at this for ages now, my GPU is GTX 1650 SUPER and I have Python 3.9.5 with tensorflow version 2.7.0 and CUDA 11.2, cuDNN 8.1.0. Here is the issue: sometimes, my models train but most of the time it errors with:

E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 

or some other variants (again, they kinda take turns fighting to be errors):

E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_MISALIGNED_ADDRESS: misaligned address E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_INSTRUCTION: an illegal instruction was encountered 

and I have absolutely no idea why its doing this. I don’t think its my code because I was able to train my code 100% of the time on CPU, but it takes like half an hour to do that and GPUs take only 3 minutes, 10 times faster!!! Who else has this error and why is it occuring randomly, also: could MSI Afterburner mess with it?