Insufficient power will not hurt a driver, however an insufficient amplifier driven into a highly clipped signal will certainly hurt a speaker. Speakers harmed from "underpowering" are done so because the system operator raises the gains on the amplifer/headunit/EQ/etc in order to compensate for the lack of output that small amplifier is capable of providing. What ends up happening is that instead of extra output, you severly clips the amplifier causing a DC waveform to be sent across the voice coil of a speaker. The clipped signal is audibley detected in the form of distortion. The problem with DC current is that an excess amount of power is dissipated in the form of heat as the driver moves a disproportionate distance when compared to the amount of power being fed into the coils. In other words, more heat is building up in the VCs than is being dissipated by the movement of the driver.
In other words, not enough clean power should never hurt a honestly rated speaker. A large amount of clipped power produded from a small, overdriven amplifier will inevitably cause damage to a speaker.