This results in some ugly scratches and dings to that wall. So, how to prevent the chairs (or any other suitable furniture) from scratching the wall? Carpet tiles!
Carpet tiles are sometimes used for kiddie play areas and may be bought in hardware stores like Ace and Handyman. Essentially they have two layers: a carpet layer and a foam layer. The carpet layer simply wipes the wall while the foam layer cushions any impact towards the wall. Just cut a couple of pieces for EACH chair. You don't need to cover up the entire width of the chair's back since it's only the two corners that will hit the wall first. Use double-sided tape to stick them on the two corners at the back of the chair as shown below.
Whenever the two chairs are pushed back when cleaning or dining, they can hit the wall but there won't be scratches. The two foam pieces are unobtrusive unless you specifically go behind the chairs to inspect them. There you go - an easy, cheap and neat solution to your wall scratch problem.