The EU isn't some black hole you throw your money into. It's obviously paid back by shared trade deals, free market and infastructure financing, among other things. We actually contribute more to the EU per capita than the UK does.
Yeah limited regulated trade deals are great and all, but regardless of how good the trade deals with the US and China are, it will still hurt tremendously due to how the system is incorporated. It's not as simple as just changing the name of the recipent, obviously.
That's also why it's guaranteed to hurt the economy on the short-term, while the long-term has the potential to be fruitful.. it's just not likely to happen in this decade.