I caught that right away and replayed it a few times. Ok Super Bowl, he came up Blind not knowing who he was shoving. I do on the other hand think he immediately figured out what he did and that is why when he turned to go after the players he seemed to have pulled back a bit. I think he was like, Oh sh!t I just shoved the ref.
Tough to toss a guy in the SB, but if you give an unsportsmanlike on shoving the ref, you have to eject, I feel that is why nothing was done, you can't justify pushing an official without an ejection.