The new USB-C standard defined at
now officially allows products to use non-USB protocols through USB ports.
Some requirements for this:
You'll need to use the USB-C connector rather than the micro-B that you mentioned in your question, but USB-C will soon be the universal standard, so this is a good idea anyway.
You'll need to provide a minimal USB interface on your device just to implement what's called a "billboard device" in case your device is plugged into a regular USB host, for the purpose of telling the host what your device is, in order to avoid user-unfriendly silent errors where you plug something in and nothing happens.
You can't directly start using your own protocol as soon as the device is plugged in. Instead, you have to do a protocol negotiation (on a separate line in the port) to indicate that you're switching to your protocol, in order to avoid confusing a regular USB host if your device is plugged into it. This negotiation would replace the proprietary handshake that you mentioned for your device.
You said your microcontroller can't handle high speed full duplex. But USB 2 runs at half duplex. Only USB 3 runs at full duplex. And for the USB billboard device that you're supposed to implement, apparently low speed (1.5Mbps) is allowed, which might be within your microcontroller and budget constraints.