the government shouldn't be allowed to gather information on people of "no security interest", but they can't know who that is without gathering information.
People can become a security interest in other ways than simply grepping bulk data. It may be justified to track convicted criminals, suspected criminals, those with links to criminals or suspects because the likelihood of them being involved is higher than a random member of the public. Likewise if in the course of an investigation you confirm that someone you have collected data on really isn't linked, then you can delete the data.
Of course you are correct to note that if you know everything about everyone you can just (in theory at least) filter out anyone and everyone who has done anything wrong and prosecute them.
The flip side is that such a data set can be misused, either if it is leaked, or by corrupt elements within the state itself.
Thus we have a trade-off - as you collect more data you create opportunities for prosecuting more crime but also for abuse. What most people seem to accept is that surveillance be used when either the confidence of the suspicion is high, or the severity is high - i.e. for active investigation of known crimes, for investigation of suspects where there is some known reason for suspicion, and (potentially) for trying to detect and pre-empt terrorism and similar.
The world is full of cost/benefit trade offs and arguments about them which assume either the cost or the benefit is infinite - people struggle to actually balance them because they are difficult to quantify.