Kell is not stupid to confront Lucifer in the Winking Cauldron. While Millie was quite visibly worried about him for sure, but it was nothing in comparison to how she reacted when Lucifer didn't show up, asking Kell if he knew what happened a damned four times a day! When the man finally did show up, she behaved as if nothing has happened, and she hasn't been literally freaking out for the last three days.

So if Kell is a tiiiiny bit irked when he finds Lucifer in the garden, that's why. Otherwise, he wouldn't have been so ... confrontational. Maybe. His mood is already shot, what else can go wrong.

"You said you had a plan! What, by the thrice cursed earth, has happened!"

He wouldn't have made a scene at his workplace, but it's perfectly fine to do one now, isn't it?