The way I understand BLE specs is that the only way the communication between 2 BLE device to be encrypted is that they need to be paired. Is that right?
This is partially correct. Pairing must occur for the connection to be encrypted the first time. However, if you store the pairing keys (i.e. bond), then disconnect and reconnect, you can use the keys that you have previously stored to encrypt the connection.
From the Bluetooth Core Specification Version 5.0, Vol 3, Part H, Section C.4, Security Re-established using previously distributed LTK.
Devices may re-establish security using a previously distributed LTK. The master device always initiates the encryption procedures, and therefore there are two possible sequences: master initiated and slave requested.
I hope this helps.