I've run into this weird issue. I found him, had already killed 2 of the 3, killed the 3rd with him, and he went off to the grove. I fought my way out and ported over to go talk to him, but when I start walking down the stairs, their is a big battle going on where all the druids are killing him. Maybe this is meant to be, but it really feels like this isn't intended.
I believe the game has a "clock" but haven't confirmed it, yet. There's a time sensitive event in the underdark that resolved itself after I took a long rest so it's possible the same is happening at that junction.