Cannot establish serial communication to ESP8266

Question

I recently designed, had manufactured, and populated my first PCB. As context for the schematic and PCB, the requirements were as follows:

1. Independent control of seven LEDs
2. The ability to program into memory specific patterns for the LEDs to turn on/off in
3. The ability to do this all over wireless (WiFi/BLE)

I am using an ESP8266 as my MCU. For several days I have been investigating a problem: I cannot establish serial communication to the UART. When I run esptool, I get the following error message:

$ esptool chip_id
esptool.py v2.8
Found 1 serial ports
Serial port /dev/ttyUSB0
Connecting........_____....._____....._____....._____....._____....._____....._____
/dev/ttyUSB0 failed to connect: Failed to connect to Espressif device: Timed out waiting for packet header 

A fatal error occurred: Could not connect to an Espressif device on any of the 1 available serial ports.

I feel as if I have tried everything I can think of. At this point I am left wondering if I have some massive design flaw in the PCB -- afterall, this is my first.

For your convenience, here are the datasheets for some of the critical components:

• ESP8266EX (MCU)
• W25X20CL (256 kB flash)

What I've tried:

• Flashing software: esptool, the Arduino IDE w/ community ESP8266, Espressif's firmware downloader, nodeMCU-flasher, etc. (I think these are all wrappers of esptool anyway).
• Host machine: A desktop and a laptop w/ Arch Linux x64, another laptop w/ Windows 10.
• USB: Three different USB cables (including one only a few cm long), several different ports across the three different machines
• Power: 12 W USB 5 V to the linear regulator input; 3.3 V rail from a 500 W ATX PSU, bypassing the regulator; adding buffering capacitors to both the input and output sides of the regulator, ranging from 100 nF to 100 uF; power over USB via FTDI USB/tty converter
• Serial: FT232 chip; CH340 chip. Added a 510 ohm resistor in series with the ESP_TX line, as per a recommendation (499R) in the datasheet. Different symbolrates (115.2 kBd, ~54 kBd, ~74 kBd, 9.6 kBd, ~250 kBd). Different combinations of bits/symbol (5/6/7/8), parity (none/odd/even/mark/space), stop bits (1/2).
• Flash: DIO/DOUT/QIO/QOUT. Tried without a flash chip even populated. Even desoldered an 8 MB chip from an old board and put that in-circuit in place of the 256 kB in my schematic. Also tried telling esptool to use different frequencies (20/40/80 MHz).
• Bootstrapping: Tried every combination of MTDO/GPIO0/GPIO2

Additionally, I've monitored the ESP_RX and ESP_TX lines with an oscilloscope. No matter what I do, ESP_TX remains idle (HIGH). However, when esptool displays the dots part of the connecting message I can decode this on ESP_RX:

c0 00 08 24 00 00 00 00
00 07 07 12 20 55 55 55
55 55 55 55 55 55 55 55 
55 55 55 55 55 55 55 55
55 55 55 55 55 55 55 55
55 55 55 55 55 c0

Here are photos are the schematics and renders of the PCB:

I apologize if they aren't the greatest quality -- this is my first time doing this.

Does anyone see a glaring issue? Obvious problem in the schematic or PCB? Something I should try in software that I haven't thought of? I'm just at a total loss here. Thanks for your help.

Answer 1

After scouring the internet for many hours, and staring at my board and schematics for many more, I've finally resolved this. Hopefully this will be of use to someone someday.

There were multiple issues.

Two of them were both critical to comms over UART:

1. MTDO (GPIO15) must be LOW during boot, for both booting from flash or UART. I had read online during design that it had an internal PDR and therefore I wouldn't have to worry about it. Despite that, as you can see I even have an external PDR (R483 -- 100k). I eventually was able to find a source from Espressif stating that MTDO actually has a weak 37k PUR. I replaced R483 with a 5k PDR.
2. The datasheet for the ESP8266EX says the minimum valid crystal frequency is 24 MHz, which is what I was using. This does not appear to be the case. Old versions of the datasheet say 26 MHz. I replaced Y1 with a 26 MHz crystal. May also be due to my crystal's stability being out-of-spec (50 ppm v. 15 ppm).

After implementing both of these changes, UART now works as expected. Uploaded programs still behave unpredictably. GPIO is unstable. This leads to the single moderate error:

1. Flash chip (256 kB) is too small. Replaced with 1 MB.

Uploaded programs now behave as expected. Wireless connectivity was the last issue to address. It works, but is very sporadic and seems to brown out the MCU, causing constant reboots. Leading to the single mild error:

1. 10-uF electrolytic added to input side of regulator.

After these changes, everything works exactly as expected.