“It’s unclear how ‘the Lord Voldemort of administrative law’ comports with our legal history and tradition,” Judge Andrew Oldham wrote in a concurrence.