LM358 datasheet says it is only able to sink a couple tens of µA when the output is close to 0V (see page 6, output current/sink).
If you check the internal schematics (page 13) this is quite obvious (Q13 can't pull the output below 1 Vbe). This isn't a true rail to rail output. It can only go to 0V if the load is something like a resistor connected to ground, in this case the opamp does not have to sink any current, so it works.
In your schematic, when the output is at 0V, the opamp output will have to sink 1.65V/10kOhm = 165µA which is too much.
You need a true rail to rail opamp, or larger feedback resistor values (which will increase noise due to bias current).
Note: LM358 was introduced in 1972. It is still produced because it works fine in its application domain, and it is very cheap. It's a timeless classic. However, real rail-to-rail opamps that actually work well are a much more recent development. Don't expect it to compare to a 50c modern RRIO opamp... also 50 cents is very expensive compared to the price of LM358...