Ooff. That was grown in a pot!
long-term survival is unlikely if they are left. I'd probably remove in stages. Start with the one right against the trunk. Then in a couple of years you can cut that one that is further out.
Then it will get to be big enough to grow into the fence.