"You've given me very little reason to care what you do or don't know."

It is the most direct answer to the question. Threatened him? With what? To report that he was trying doorknobs in some quiet corridor where no one was meant to see. Hardly a high crime, he suspects. Maybe he was tired. Maybe he was after a quiet place to be alone. The list of possible lies he might tell to explain himself must be at least as long as the one this stranger might produce to indict him.

"Now hurry along, won't you? Surely we both have better things to do."