~/.ssh/config に AddKeysToAgent yes を書くと GitHub などでの複数アカウントログインに失敗する

結論

表題のとおりなので、同じホストに対する設定内容で AddKeysToAgent yes と書くのはやめましょう。

参考

$ man ssh_config
（省略）
     AddKeysToAgent
             Specifies whether keys should be automatically added to a running
             ssh-agent(1).  If this option is set to yes and a key is loaded
             from a file, the key and its passphrase are added to the agent
             with the default lifetime, as if by ssh-add(1).  If this option
             is set to ask, ssh(1) will require confirmation using the
             SSH_ASKPASS program before adding a key (see ssh-add(1) for
             details).  If this option is set to confirm, each use of the key
             must be confirmed, as if the -c option was specified to
             ssh-add(1).  If this option is set to no, no keys are added to
             the agent.  The argument must be yes, confirm, ask, or no (the
             default).
（省略）