That is not a trivial project. There are many educational sub projects to worry about. One is buttons and debouncing. Another is writing characters to a display. There is the decision of how you want to implement this are you interested in making it from a big box of nand gates or want to take a microcontroller or other processor and write software? Are you interested in using an fpga and doing all of the math in RTL? You need to break the problem down into those components and work/learn one component at a time and then join them together. For example if the core math engine is actually software on a microcontroller, one task would include writing some C functions on your desktop computer that you can feed keystrokes and output characters that will ultimately go to the display. A non-trivial task if you have never programmed before. If you are choosing to go with an fpga or possibly even discrete logic, you might want to use verilator or icarus verilog or ghdl to work on the core math and interface modules.
As an educational project what I would do is get a few msp430 launchapads, under 5 bucks each or The STM32 value line discovery (the stm32/arm based one not the other one) for around 12 bucks each. A number of folks will steer you toward the arduino, and that is a fine platform as well, it has its pros and cons, i wouldnt go with it as my first microcontroller. Buy a simple two line lcd panel, earth lcd used to be a good place, perhaps just go to sparkfun. Take one microcontroller board hook it up to the lcd panel and learn how to put characters on the display. I would then learn how to use the uart on the microcontroller which often starts with blasting bytes out, then later receiving and echoing. Use the uart receiver to receive things to put on the display then using a dumb terminal (putty, hyperterm, minicom) from a computer feed stuff in and make sure it works. Next take another microcontroller, use your uart in and out experience and work on the core math engine, from your computer feed it 0 - 9, +, -, = at first then add multiply and divide and then floating point if you are brave enough for that (or have a library that fits). Output from the math module would echo input numbers and print results when = is sent in, etc. Then figure out what to do with buttons, find an array of buttons, feed those into the third microcontroller somehow, debounce, and have that turn into uart out of 0 - 9, + , - , = to the math microcontroller. THEN, reduce all of this into a single microcontroller without the uart stuff in the middle.
Another alternative is to get one of the rs-232 fpga boards from knjn.com or the lattice brevia (is that big enough?) or a number of others, then work on each of the functional blocks using an RTL language. parts of it will be much easier than the equivalent software solution, some parts will be a bit harder than a software solution.
If you can provide more information as to what you are thinking, a box of nand gates or microcontroller based solution or were you thinking about something else?