Ya know, that piqued my interest in something related to recording every minute of your waking life in video, and I think this is more reasonable, useful, and feasible today.
1. Record audio 24x7. I'm not even going to bother running the numbers - it's a hell of a lot smaller, and the next step means we don't need to calculate out that far.
2. Transcribe speech-to-text.
That's it. That could be done on a nearly any phone today. Could chunk the audio into 1hr segments, or daily, or whatever works for you. I'd probably do daily, doing the transcription at night possibly?
Then you'd have a text file per day of all words you heard or spoke. Forgot what time someone said to meet them? That should be trivial to grep for out of yesterdays file.