‘So you can’t pick and choose what day you like a judge and what day you don’t!’ Trey Gowdy declared on Monday