Question - "As long as the card is exiled, you may cast it."
So, once I cast the exiled creature exiled by my hostage taker, the opponent cannot kill hostage taker and get it back. The creature is now under my control and hostage taker becomes a 2/3 vanilla. Is this correct?

So powerful! It's almost a mind control plus a free 2/3 body if opponent doesn't answer immediately.

you are correct, although note that you still have to pay for the spell in order to cast it from exile, so it's not quite as efficient as mind control when you're trying to take expensive things.