I drive on tracks, not the bush, and I leave the areas I go to better than I left them meaning I take away other people's rubbish if required. I take my own fire wood and don't put plastics in the fire. I don't do waste water (I have a grey water tank), and if I have to road build I make it right before leaving. Other stuff as well.

But you said it well, the off roading I do is of no consequence when compared with what most humans do...flicking cigarette butts and rubbish all over the place, overusing fossile fuels, general wastefulness, consumerism, new phones every year, use of chemicals...the list is long.