After OP cleared the question, I think that it would be useful to clarify something about the main ways of programming an Atmel microcontroller:
So, one of the main differences between serial programming and SPI programming is: for serial programming, the μC has to have a bootloader.
Serial programming = four wires, two for data: TX and RX. Needs a bootloader.
SPI programming = six wires, four for data: Doesn't need a bootloader.
MISO, MOSI, Clock, Reset
There are some USB programmers (green box in the table above) that looks like a USB-to-Serial adapter (orange box in the table above), but each one works in a different way.
It seems that OP has used a programmer before (option 3 in the chart), and now OP is trying to use a serial adapter (option 2 in the chart). But this option needs a bootloader.
(edit) But, for several devices, the serial option is not adviseable, because the bootloader ocuppies a lot of memory space, and for simpler chips (this is the case with all ATTiny) there is no much space left available for the user's program.
The final answer to OP: for programming Attiny2313A with the Arduino IDE and Atmel ICE, you need to connect 6 wires from Atmel-ICE to Attiny: VCC, GND, and the four wires of SPI interface (MISO, MOSI, CLOCK, RESET). You will need to load some drivers and libraries to the Arduino IDE, in order to Arduino IDE can control Atmel-ICE for burning the code to Attiny.
For doing this, you will not need to use RX and TX pins of the microcontroller.
Don't worry about the absence of a capacitor in RESET line. It is good, but not mandatory.