Keep it encrypted on your $MEDIA. You keep multiple Yubis for access.
That's a solution looking for a problem, or more for creating a problem. It's a very common use case for people who want to stay sane enough while trying to unnecessarily shoehorn Yubikeys in their workflows, but these really aren't made to just keep a regular secret symmetrical encryption key to your backups or password manager. You CAN use some feature to achieve that but it's pointless as the host machine sees the secret data and is doing all the decryption. You are better all around by just using a regular password here.
The whole point of these cryptographic tokens is that they're themselves different machines, air gapped (or if you want connected through a dedicated and very limited interface) from the computer/phone that runs a general purpose OS that can be compromised. This comes with great inconveniences, like you can't multiply them (so you need to register multiple keys separately on all your services, and some don't accept multiple passkeys/FIDO - PayPal I'm looking at you), you can't back them up, they have limited capacity and so on. But once you go "darn it, I'll just do everything on my computer in a password manager" you can't roll that back by putting some access control based on Yubikeys, it does nothing, it's still everything done on your computer.