currently there is an assert to prevent this, we should change the implementation to make sure this works well