Question
How to troubleshoot a BLDC motor?
Answer
Short Answer - Last Update 2020aug25hkt1609
Part 2 - Case Study 2 Nidec BLDC Motor
Now I know how the BLDC controller looks like:

(7) AliExpress Nidec BLDC 12V 4800RPM PWM 24H677H010
DC12V no-load speed 4800 rpm, no-load current 180mA, stall current to 2A
Pin definition:
Red: 12V positive, the motor drive power positive DC7.5V-12V
Blk: the power supply negative, the power supply negative
Ylw: emergency stop, brake, short supply power supply negative brake
Wht: PWM 20-30 kHZ
Grn: positive and negative, positive and negative control, low level reversal
Blu: NC
Brw: square speed signal, 6 pulses per revolution
Orn: NC
Part 1 - Getting to know GM3525 BLDC Motor
I found the following motor in my junk box. I guess it must be a BLDC, becauseit has five coloured wire, though not exactly the same as the OP or the other JBG37 motor I have been reading about. It is labllled GM3525. I guess JGB37 is 37mm diameter, the OP's motor should be the better JB54, or 54mm diameter, and the one I am testing should be 25mm.

The OP says he new motor is not moving. So my first step should be apply voltage and get it moving.
Part 1 - Wiring the GM3525 BLDC Motor
The GM35 motor connector has only 5 wires, but GB37 has 6 wires. I found that GM25 doesn't have the green wire, which is the encoder ground. So I guess this smaller motor should have merged all together motor ground, encoder ground, and encoder signal. Anyway, I think I will start the motor power to see if it can move. GB37 spec says the power is 6V to 24V. I guess newbie me should play safe and start with the lowest 6V.
[BLDC] 3525 12V/24V Motor with encoder ¥80, TaoBao
Introduction

This is a continuation of the long answer below. The long answer is only for newbies like me who knows very little about BLDC motor. Those already have a basic knowledge of BLDC motor might just skim my long answer below and start reading the short answer here.
This short answer focuses on the BLDC motor controller. The main goal is not to dig into the details of the controller, but on how to wire the 6 pin connector, which is what the OP asks and needs to know, before trying to troubleshoot his new BLDC motor he finds not working.

Those who want to get some professional level knowledge about the BLDC controller theory and operation, I would recommend to read the following introduction tutorial by the Analog Devices engineer C Frick.
An Introduction BLDC Motors - Charles Frick, Analog Devices
For those impatient newbies who want to start setting up their drone BLDC right away, I recommend to at least read the following must read from AdaFruit. This must read actually clarifies the OP mind on the confusion between BLDC and ESC, where is the controller etc, ...
An introduction to Brushless DC Motor Control, Bill Earl, AdaFruit
The following is extracted from the AdaFruit tutorial
(1) Brushless DC motor control is not quite as simple as the regular brushed type. Brushless motors require a controller that can sense the proper time to reverse the voltage to the coils.
(2) Sensor or Sensorless? - Some motors have built-in hall-effect sensors that can detect the orientation of the rotor. Controllers for this type of motor require sensor inputs to read these hall sensors.
(3) Other 'sensorless' motor controllers sense the back EMF in the motor coils themselves to detect the rotor position. Most ESC (Electronic Speed Controls) for brushless RC motors use this technique.
(4) Integral or Separate? Many BLDC motors have integral controllers. Computer fans are one example. These may or may not have PWM inputs or tachometer outputs. If speed control is one of your goals, be sure that you understand the capabilites of the controller.
(5) Brushless motors designed for autonomous and remote control aircraft and vehicles typically require a separate controller. These are typically of the sensorless type and use standard servo type pulsed signals for speed control.
I must confess again I am only a BLDC newbie, so I am not sure at all my reading recommendation above is good. Actually I don't bother to understand too deep the technical article, because I know the BLDC controller is embedded inside the motor, all I need to know is how to wire the connector.
However I will be going to do the real thing, wire a real BLDC motor and try to drive it using Rpi4B python. Stay tuned.
to continue, ...
Long Answer
Part A - TLDR
All my life I have been playing with toy DC motors, and I know them very well. However I have almost zero knowledge and experience on BLDC motors. I know that drones use BLDC motors, but actually I never owned any RC toys in my life, not to mention RC drones.
I also know quite a lot about 5V/12V regulated power supplies for toy motors, but I don't know what is ESC, and I have never seen the real ESC thing in my life.
I need to introduce my sub newbie knowledge and experience on BLDC motor because I must warn the experienced RC guys that I am ignorant in RC, so bear with me talking stupid RC stuff. RC Ninjas can skip the following parts basic parts and start with the real BLDC thing.
But I would appreciate it very much if anyone can point out my misunderstanding on RC, ESC, and BLDC, so I can make corrections ASAP, before letting EE SE losing too much face, ...


Part B - Googling and Wikiing on BLDC Motoers
So I wikied and sursprisingly found that BLDC is much more complicated than the stupid DC motor, because inside it there is a switching power supply providing AC current, and most amazingly is that there is a control loop driver controlling the speed and torque of the motor.
This is indeed exciting, because I am starting to learn some control engineering theory and practice. Now I can boast to my bad friends that I am not just an plain vanilla EE hobbyist, but also a computing and control hobbyist. So my bad Anono friends should from now on respect me much more than I deserve, ...
Usually my bad friend would not trust me and always need to verify my asking questions I don't know how to answer, so I usually defend myself by attacking questions first, say "Do you know why BLDC are so powerful, because they ...

Part C – BLDC Motor, A case study
I found that the OP’s BLDC motor is a bit expensive, and also high speed, high torque (5,000rpm!). I think this scary 5,000 rpm motor is too dangerous for me newbie to handle. So I searched AliExpress for a poor hobbyist affordable (US$20), and also newbie save gear model with a speed range of 10 rpm to 500 rpm. The most important thing is that the both the OP’s expensive and my cheap motors are using the same BLDC standard of 6 pin colour coded connector.
AliExpress JGB37-3525 37mm Gear BLDC Motor US$20 – Aslong
Amazon JGB37-3525 (12V – 33RPM) 37mm Gearbox BLDC Motor US$22
Both AliExpress and Amazon are selling at the same price of US$20. I think Amazon is recommending the 33rpm model because it is safe for newbies, also the maximum (stall) current is only 0.7A. So it should be safe to use 12V Lipo power bank to do the tests.
bldc current spec

The OP is curious to know what kind of driver is inside the BLDC motor and will the driver clash the ESC. Now I know the built in controller in the BLDC guy is a high class feedback controller, and I guess the ESC guy is as most as clever the toy servo. So I am no more interested to look into the ESC thing (Actually I should pretend not not have heard of ESC, so my bad friends would respect more that I deserve)
Now I am digging deeper into BLDC. I am happy to find from the Amazon that their motor is teardown/hacker/newbie/prototyping/enggrdesign friendly, eg easy to change speed by replacing different speed (or fried, clashed, I mean) gear box.
bldc gear

Another good thing about the motor is that the specification is quite professional, eg with detailed engineer drawing, showing things I am interested, eg, the shaft diameter is 6mm. I used to play with Mabuchi TT130 4WD type motors with a miserably thin/weak 3mm shaft diameter, or bigger motors with 4mm,, 5mm diameter. Now this motor is heavy duty 6mm. So I should now upgrade my motor status from newbie to ninja, and from now talk less to the low tech DC only, non BLDC guys, so not to damage my reputation
The OP is asking how to wire the following connector, to make the motor moving.
Now let me see if the AliExpress/Amazon BLDC motor is using the same connector.
I first looked at the OP’s connector and found it not identical to the BLDC motor I am study. So I went to check out the other motor, and found the following spec:
jgb37 connector

I very surprisingly found that the reason why there are as many as 6 wires, in two groups:
Motor power = 2 wires,
Encoder = 4 wires, 2 for power, 2 for encoder A, B phase.
What I found interesting is that there are two power wires, but no PWM signal control wire. My first guess is that the motor +ve, -ve power wires when go inside the motor casing, can be switch polarity to select CW and CCW. This is indeed strange, is there a full bridge inside the “controller”. I am as curious as the OP, and I should tear down the controller inside black cylinder, to see what is going on there.
Ah, locking down supper time. Call it a gloomy Sunday. Carry on till tomorrow.

/ to continue, ...
References
(1) 38SRZ-S BLDC Motor 12V/24V 2000/3000/4000/5000 rpms with built in drive for DIY US40 - AliExpress
(2) BLDC (Brushless DC) Electric Motor - Wikipedia
(3) An introduction to Brushless DC Motor Control, Bill Earl, AdaFruit
(4) [BLDC] 12V Motor with 64 CPR Encoder for 37D mm Metal Gearmotors - Pololu
(5) [BLDC] 3525 12V/24V Motor with encoder ¥80, TaoBao
(6) Nidec BLDC Motors Tutorial
(7) AliExpress Nidec micro DC motor 24H677H010 brushless motor 12V 4800RPM brushless PWM motor
/ to continue, ...
Appendices
Appendix A - BLDC Motor Wiki Reading Log
(2) BLDC (Brushless DC) Electric Motor - Wikipedia
So I wikied and learnt a lot, and made a reading log, which is a summary of the introduction part of the wiki column.
A BLDC motor, also known as electronically commutated motor, and synchronous DC motors, are synchronous motors powered by DC current via an inverter or switching power supply which produces AC current to drive each phase of the motor via a closed loop controller. The controller provides pulses of current to the motor windings that control the speed and torque of the motor. This control system replaces the commutator (brushes) used in many conventional electric motors.
The construction of a brushless motor system is typically similar to a permanent magnet synchronous motor (PMSM), but can also be a switched reluctance motor, or an induction (asynchronous) motor. They may also use neodymium magnets and be outrunners (the stator is surrounded by the rotor), inrunners (the rotor is surrounded by the stator), or axial (the rotor and stator are flat and parallel).
The advantages of a brushless motor over brushed motors are high power-to-weight ratio, high speed, electronic control, and low maintenance. Brushless motors find applications in such places as computer peripherals (disk drives, printers), hand-held power tools, and vehicles ranging from model aircraft to automobiles, ...
End of answer