I have written plenty of bare metal code for PIC and x86 processors. Can someone tell me how and when should I need an operating system? Conversely, what application or situation can be dealt with or without an operating system as well?
My rule of thumb is that you should consider an operating system if the product requires one or more of the following: a TCP/IP stack (or other complex networking stack), a complex GUI (perhaps one with GUI objects such as windows and events), or a file system.
If you've done some bare metal coding then you're probably familiar with the super-loop program architecture. If the product's firmware requirements are simple enough to be implemented with a super-loop that is maintainable (and hopefully somewhat extensible) then you probably don't need an operating system.
As the software requirements increase, the super-loop gets more complex. When the software requirements are so many that the super-loop becomes too complex or cannot fulfill the real-time requirements of the system then it is time to consider another architecture.
A RTOS architecture allows you to divide the software requirements into tasks. If done properly, this simplifies the implementation of each task. And with task prioritization an RTOS can make it easier to fulfill real-time requirements. An RTOS is not a panacea, however. An RTOS increases the overall system complexity and opens you up to new types of bugs (such as deadlocks). As an alternative to the RTOS you might consider and event-based state machine architecture (such as QP).
If your product has networking, a complex GUI, and a file-system then you might be at the point where you should consider full featured operating systems such as VxWorks, Windows, or Linux. Full featured operating systems will include drivers for the low-level details and allow you to focus on your application.
It really depends on your definition of an 'embedded system'. There may be some who would claim that if it isn't bare-metal programming, it's not embedded (which precludes your question), but I would disagree with that - I would argue that any system which is designed to perform only one function, i.e., to only run one specific 'application', could be called an embedded system.
That said, it should be fairly easy to imagine situations that would benefit from the services of a full blown OS. For instance, where I work it is quite common to find people building test equipment on top of an instrumentation design suite that runs on top of windows. These systems are configured to boot into the test station configuration and lock out general usage (to prevent corruption of the station) and are arguably therefore embedded systems.
However, just buying off-the-shelf I/O modules, plugging them into a rack mount PC, and whipping up a configuration in a GUI may fail to qualify as embedded system design to some. For a little less off-the-shelf situation, consider a custom process controller with an FPGA, for which you want to do some fancy data logging. You might embed a soft-core processor system (with an existing BSP) and run a realtime linux in order to run a network stack (for your logging and NTP etc) and do everything else in logic.
My (very vague) rule of thumb is if you need more than one thread of control (say at least one device that involves a protocol or a state machine plus something else to do) then some for of OSish software will make your life easier
An old question but I'll comment anyways.
Even if you don't have network stacks or similar, at the point where you need a task scheduler as you have enough processes in your embedded application, you might consider a RTOS. Writing a simple timer-based cooperative multitasking scheduler isn't that hard but making sure a stuck process won't block the rest of the application and such can take a while to get right. You need to implement a priority system with some kind of provision to bump processes down in the queue if they have not completed.
RTOS also gives you things like memory protection and the like which makes it much easier to track some common gaffes in C code but simple microcontrollers simply may not be able to handle complex memory protection. E.g. MSP430 allows you to separate code and data in high level but there's no fine grained memory access control.
An operating system actually bridges the gap between hardware and application software (through device driver). In other words, it provides relatively high-level platform for the programmer which ultimately reduces the code complexity. And further, the operating system provides strong and flexible platform for the execution of application.