The general problems with this relate mostly to confusing the user, especially with the breaking of the back-button (so that in a new page they can't press back to get back to where they were). It can be terribly annoying to close a page that a link popped up instead of doing the natural intuitive thing that has been learnt, and it will make users generally distrustful of your site.

In my own opinion one of the exceptions to this (although a lot would argue) is zooming-in on pictures. This is because there really isn't any nice intuitive, simple, non-intrusive way of doing it (that I've found at least). As long as the enlarged image takes up less than the screen and is positioned so that the user can still see most of the original page so they're not wondering where their back-button's gone, then I think it's reasonable.

Snow