Solution for me was to delay the destruction of the portal. This simple function did the trick:
private IEnumerator WaitBeforeDestroy(float wait)
{
yield return new WaitForSeconds(wait);
Destroy(gameObject);
}
On second thought: Just adding Destroy(gameobject,2f) would probably do the same.
The problem seemed to be that the portal was destroyed before the character position was properly updated, so delaying the destruction seems to do the trick.