UX Idea: Elevator Buttons

I have a confession to make. I am that asshole that accidentally pushes the wrong button more often than I would like to admit. Whether it is the up/down buttons or somehow the wrong floor, I hate myself when I do it and I feel like a complete idiot.

A simple solution for a simple mistake. You push the button to select your choice... why can't you just push it again to unselect your choice?