Does differential impedance between D+ and D- not only depends on separation between and wire gauge of D+ and D-, dielectric constant of the insulation, but also the GND wire and shielding?
The Characteristic impedance of a wire (more correct: transmission line) depends on the (signal) conductor size and what other conductors are nearby. So yes, insulation and shielding influence the Characteristic impedance.
But I sense that you're thinking that you can influence that Characteristic impedance by (not) connecting the shield to ground. That is not the case! The Characteristic impedance is a property of the wire itself, irrespective of how you connect it.
If the Characteristic impedance of the impedance controlled lines and the USB cable are the same and you're able to make a "nice" connection between the two then that can work and you must connect the shielding of the USB cable to ground because that's how it is supposed to be used. If you leave the shield floating the capacitance between shield and signal lines will cause extra coupling between the 2 differential signal lines. Grounding the shield will minimize this effect.
However, unless you're going to use this at a very low speed I doubt that this is going to work properly as I am unsure that the Characteristic impedance of a USB cable and PCIe lines are the same. Also connecting the USB cable to those PCIe lines without making it such that signal reflections will be an issue, will be a challenge.