There are many devices that perform various cryptographic operations inside their microchips. One of their key feature is that the secret keys are stored inside of the hardware and never leave it. Since it is possible to reverse engineer a chip, I am wondering if it is also possible to get at the chip's internal memory as well to steal the data stored in it?
In general yes that is possible. For example a guy called Christopher Tarnovsky managed to tap the inner data paths of a Trusted Platform Module. He used acid to remove the encapsulation, rust remover to get rid of some mesh and then managed to probe the circuit and intercept internal communications.
Obviously on such a small scale it is a non-trivial excercise, but there is no theoretical reason I'm aware of why you couldn't disconnect the address and data bus of an on-chip ROM for example and read back the contents once you have physical access to it.
Nothing is 100% safe.
That being said, the time and effort needed to 'crack' a chip can range from trivial to exponentially beyond feasible.
I personally like how Microchip summarizes the matter at the beginning of their PIC and dsPIC datasheets (especially the last bullet point - emphasis mine)
Note the following details of the code protection feature on Microchip devices:
• Microchip products meet the specification contained in their particular Microchip Data Sheet.
• Microchip believes that its family of products is one of the most secure families of its kind on the market today, when used in the intended manner and under normal conditions.
• There are dishonest and possibly illegal methods used to breach the code protection feature. All of these methods, to our knowledge, require using the Microchip products in a manner outside the operating specifications contained in Microchip’s Data Sheets. Most likely, the person doing so is engaged in theft of intellectual property.
• Microchip is willing to work with the customer who is concerned about the integrity of their code.
• Neither Microchip nor any other semiconductor manufacturer can guarantee the security of their code. Code protection does not mean that we are guaranteeing the product as “unbreakable.”
Code protection is constantly evolving. We at Microchip are committed to continuously improving the code protection features of our products. Attempts to break Microchip’s code protection feature may be a violation of the Digital Millennium Copyright Act. If such acts allow unauthorized access to your software or other copyrighted work, you may have a right to sue for relief under that Act.