In some circumstances, I suppose that you are correct that verbal, emotional, or physical abuse is crueler.
An affair is just another form of emotional abuse, or it may a reaction to abuse.
If one wants to follow their own values, though, they should file for divorce first.
Twenty years later, I received an apology from my ex for her behavior like all of the above. I stayed married the last few years to protect the kids.