Defining 'art' conclusively is functionally extremely problematic, or arbitrary. There are certainly, however, things called 'art'. Therefore we can only argue about 'things called art', which are objects.
Art is personal property. We should not destroy or damage others' personal property, therefore we should not deface art. Unless you own it. Then do what you want.
There is public and private property. Public property is still not yours to destroy. You can't go pulling up roads and claiming that they're yours so you can do what you like with them.
I never mentioned legality or ethics or not, but, yes, destroying or modifying someone else's property without their prior consent is something that people shouldn't do.
You could be losing the original intent of the artist, lose a potential source of inspiration for artists in the future, as well as a record of our artistic history.