Is the People Pane secure?

Some Microsoft Outlook users are concerned that the People Pane is sharing their private email with other users.