I am working on a bandpass filter to filter the output from a hydrophone. This is my first time designing a filter and using a cookbook from TI, I built a 4th order Butterworth filter with two partial 2nd order MFB filters. The filter is working as expected when tested in ltspice but the rolloff is not as steep as I want it to be. To get a steeper rolloff, I am thinking about cascading 2 or more of these 4th order filters and create higher order ones. Is that ok or are there any tradeoffs for doing so, other than the increased complexity of the circuit? Will the PCB layout difficulty increase? Are there any better approaches to get both a flat response at mid frequency (the bandwidth is only 10% of the mid frequency) and a good rolloff? Is using a Chebyschev optimization instead of Butterworth a better idea for my use case?
To get a steeper rolloff, I am thinking about cascading 2 or more of these 4th order filters and create higher order ones.
Cascading two 4th order Butterworth filters does not give you an 8th-order Butterworth filter.
If you want the benefits of a Butterworth filter (maximum passband flatness), then you should go back and design an 8th-order Butterworth filter.
Is that ok or are there any tradeoffs for doing so, other than the increased complexity of the circuit? Will the PCB layout difficulty increase?
More components means more PCB area.
As another answer points out, more stages means more noise.
Is using a Chebyschev optimization instead of Butterworth a better idea for my use case?
A Chebychev filter will let you trade off passband ripple for steeper roll-off. Presumably in the limit that you specify minimum ripple you will just get the Butterworth filter (but I'm not 100% sure on this point).
Only you can decide what is the correct balance between roll-off and ripple for your application.
Are there any better approaches
If you are targeting some well known application where there are many customers who need the same type of filter, there may be a drop-in filter component available to suit your needs. This could be a passive resonant structure that can act as a high-order filter in very little board area, and with no noise sources. The suitable technology and availability will depend on your exact center frequency and pass-band requirements.
The main thing is the number of amplifiers that must be used to create the filter. In general adding more amplifiers will add more noise. Do a noise analysis in spice to get a basic idea for what the noise is (and be careful because some manufacturers models do not accurately reflect real world noise values or noise values found in datasheets).
If SNR is a concern then the noise added by filtering must be considered.
Other than that, adding stages to a filter and adding amplifiers will also increase the obvious design parameters as you've mentioned like PCB area, power consumption, ect.
Because of tolerance issues in capacitors, high-order multipole filters are notoriously difficult to actually build, despite working well on paper.
For this reason, the industry provides us with filter building blocks that can be tuned entirely with resistors. The UAF42 is one such device. This makes filter design much easier.