"Dark things! It's how the madness starts! Seeing things and hearing voices!"

This is not a moment to be calm and Mat doesn't like being told he ought to. He's begun to lose his mind, he cannot be calm about this. There is no quiet acceptance or trusting himself to something greater, he's losing his mind and he's not about to do so without a loud protest.

Besides. Yennefer promised. He doesn't want to hear 'calm down', he wants to hear 'I can fix this'. He's so wrapped in his fear that he doesn't even have the time to note how much nicer her room is.