The visitors made it 11 unbeaten after ending United's 10-game home streak without defeat.
Alex Penny lost Kyron Gordon to head Harriers ahead before Jake Wright jnr levelled.
However, Ashley Hemmings' drive took a wicked deflection to catch out young keeper Marcus Dewhurst with 14 minutes to go.
"I'd argue it's probably the best we've played all season and football can be cruel at times," Elliott said.
"That's the one thing, over the course of the season, we are making individual errors (Gordon fore the opener).
"It's not the same player all the time, we conceded a similar goal last week.
"They're mistakes we've got to cut out but the performance in general, I thought was fantastic."
Teammates Luke Shiels and Scott Garner rushed to Dewhurst at full time to console the keeper, who felt he should have dealt better with the freak deflection.
"Marcus is disappointed with the second goal. It's a strange goal," Elliott said.
"I thought it had gone over the bar to be honest. Difficult to accept but it's hard to be critical(of the players)."