Re-write remove_prefs to not use recursion.

This also fixes the possible use-after-free that scan-build seems to
think is there.