Pre-VAR, it got so difficult for refs to officiate a game because players were throwing themselves to the ground without contact, feigning injury, screaming in pain and surrounding and bullying refs when they don't get their way. Even corners and throw ins getting appealed and arms thrown up when it went the other way.
And all that still goes on.
What little authority the refs had is diminished by them avoiding decisions or making bad ones in the knowledge that VAR is there.
Sort out that kind of cheating with all refs on a microphone during the game and post-game video analysis and bans for cheating and ref abuse, if needs be, and a lot of the controversy goes away.
You can then use VAR for 20 second maximum further review to assist the ref for clear errors on offside, handball, out of play, etc.
Subjective stuff like whether a tackle is no foul, a foul, a yellow or a red, I would leave to the on-field ref during the game.