What the president was required to do was simply in execution of the act of Congress. It was not the making of law. He was the mere agent of the lawmaking department to ascertain and declare the event upon which its expressed will was to take effect.
-Field v. Clark
Congress can take back the authority, but this one won't.
I suppose you can argue that the Supremes are your "enough judges," but that comes a bit late, it was decided over 100 years ago, long before Trump. The ability of Congress to delegate its powers is also the basis of all regulatory law. Congress didn't pass a law requiring headlights on cars, that's a law created by non-elected regulators in the Executive branch.
A conclusion is simply the place where someone got tired of thinking.