At the vs With the

My heart grieves with the thought that he will stay back.
Vs
My heart grieves at the thought that he will stay back.

Grammarly corrects the first sentence to the second one. However, I have come across the first structure a lot more than the second. At the thought and With the thought—can these two be used interchangeably? Or do they mean different?

Both sound OK to me. To portray a direct reaction to something, "at" might be better (e.g. physical reactions like "I jumped at the noise", "I laughed at the joke"). To portray the effect of contemplating a topic, "with" might be better. Emotions can often be portrayed either way.