[SOLVED][UPDATE]:
The problem was with fuse bits. Somehow the fuses were being incorrectly written as 0xFE rather than 0xFD. FE corresponds to 1.8V and thus arduino board was working down to 1.8V. I reinstalled Arduino and started everything from scratch and with the Show Info code suggested by bigjosh, I was able to verify the fuse bits. I set the BOD level to 2.7V and now it's working as expected. Micro-controller resuming all operations once sufficient voltage is available.
Another update: I was seeing extended fuse settings for 1.8V because of file write permissions on C:\ in windows. Apparently the boards file was read only and windows raise no errors when I hit Ctrl+S. It led me into thinking that boards file was being updated while it was still stuck at default 1.8V BOD level. Second time, I chose to install on D:\ and everything is working perfectly.
I am trying to figure out how to use BOD to make my arduino project more robust but it appears to be yet another trouble. My system runs on 3.3V so I chose to set the BOD level at 2.7V (Extended fuses = 0x05).
Just as a bench test, I uploaded a serial print code which print "Started" when the micro-controller first starts and then "Working" after every half seconds.
Experiment was to power up the atmega using a bench top power supply and monitor the serial text on a PC. I was expecting something weird to happen once voltage goes below 2.7V. Nothing happened until 2.2V. Below 2.2V, multiple scenarios were there:
(Experimental setup: I didn't use arduino board. I built it on a separate breadboard and voltage was directly being fed by using bench top power supply. There were filter caps - 300 uF 16V close to IC pins. Serial was connected to PC directly without optical isolation. Common GND was made as well.)
Case 1: Micro-controller resetted a few times and kept working normally at 2.1V or so. On further reducing, it stopped printing. On raising the voltage, it never resumed to a normal working state. External reset brought it to normal state.
Case 2: It kept on printing some random stuff on serial until I further lowered the voltage, at which point it stopped printing. It didn't resume printing even after voltage was brought up to normal levels. External reset brought it to normal state.
Case 3: It resetted a few times and kept on printing "Wmrkmng". At that point, I started to raise the voltage to normal levels (3.3V) but it still kept on printing wrong text. External reset brought it to normal state.
My expectation was that when brown out is detected, the microcontroller would shut down (all GPIOs LOW and UART not printing anything - similar to powered off microcontroller) and as soon as voltage becomes normal, it'd act as if it has been turned ON from an OFF state. However the results are totally different. As such, if I use it in a product and for some reason, brownout occurs, the only convenient way to get it into a working condition is to power it OFF and back ON so that it receives Power-On-Reset signal. Definitely such a procedure won't be acceptable in many cases (including mine). As such what should I do to prevent such a lock up state? I am not sure even watch-dog will be of much help (consider case 3 - Most of the code was working. What if WDT sees no issue and lets the micro-controller run the way it is running).
(Edit: WDT failed in a few cases as expected)
I couldn't find much in Atmel datasheet but I came across a stackexchange question here which describes a similar issue for PIC family.
(Edit 2: While I waited for any answers to this question, I performed some other tests which didn't involve any external connection at all. I removed serial and uploaded a simple code to blink an LED at 1 Hz when a pin was HIGH and at 5 Hz when the pin went LOW. Even this code didn't work out. The results in this case can be compared to case 1,2 and 3. I tested WDT as well. No benefit. One strange thing that I noticed was that even though I was changing the bootloader config to set different BOD levels - 1.8V, 2.7V and 4.5V, the code used to mess up at around 2.2V in all cases). Here is the arduino definition:
uno.name=Arduino Uno
uno.vid.0=0x2341
uno.pid.0=0x0043
uno.vid.1=0x2341
uno.pid.1=0x0001
uno.vid.2=0x2A03
uno.pid.2=0x0043
uno.vid.0x2A03.warning=Uncertified
uno.upload.tool=avrdude
uno.upload.protocol=arduino
uno.upload.maximum_size=32256
uno.upload.maximum_data_size=2048
uno.upload.speed=115200
uno.bootloader.tool=avrdude
uno.bootloader.low_fuses=0xFF
uno.bootloader.high_fuses=0xDE
uno.bootloader.extended_fuses=0x05
uno.bootloader.unlock_bits=0x3F
uno.bootloader.lock_bits=0x0F
uno.bootloader.file=optiboot/optiboot_atmega328.hex
uno.build.mcu=atmega328p
uno.build.f_cpu=16000000L
uno.build.board=AVR_UNO
uno.build.core=arduino
uno.build.variant=standard
I changed extended fuse bits to 0x04, 0x05 and 0x06 to test the three BOD levels.
Basic LED blink code used to test:
int ledPin = 17;
int inputPin = 2;
void setup() {
// put your setup code here, to run once:
// Serial.begin(115200);
// Serial.println("BOOT");
pinMode(ledPin,OUTPUT);
pinMode(inputPin,INPUT);
}
void loop() {
// put your main code here, to run repeatedly:
// delay(500);
// Serial.println("XXXXXXXXXX");
if (digitalRead(inputPin)==1)
toggle(17,1000);
else if (digitalRead(inputPin)==0)
toggle(17,200);
}
void toggle(int pin_number, long int delay_time)
{
digitalWrite(pin_number, HIGH);
delay(delay_time);
digitalWrite(pin_number, LOW);
delay(delay_time);
}