Fix the issue where the contact list would sometimes duplicate everyone

Testing Done:
This was tested with another review request that was reproducing it 100% of the time.

Reviewed at https://reviews.imfreedom.org/r/2431/