Back when Kubernetes announced the new StatefulSet feature with K8s v1.5 (converting it from the old PetSet name), they put out a really good blog post walking through an example of its usage: https://kubernetes.io/blog/2016/12/statefulset-run-scale-stateful-applications-in-kubernetes/
The first paragraph has a really good description of differentiating features that always stuck out in my mind:
...“at most one pod per index” for deployment of the Pods in the set. Along with ordered deployment, ordered termination, unique network names, and persistent stable storage ...
Break down the important ones:
Unique network names
In a deployment, it doesn't matter the network name as any pod should be able to pick up the work. However, when you are running things like a stateful cluster, you might have the concept of 'masters' and 'slaves'. This unique network naming will allow you to keep the distinction by referencing a specific pod, with a predictable incremental name.
Ordered Termination
This is important for re-deployments in a stateful cluster as you could have critical machines that need to stay up longer. If you have coordinator or 'leader' nodes, those should be the last to be restarted. StatefulSets allows you to do this. The linked blog post even shows an example of this.
Persistent stable storage
Arguably, this is the most crucial difference. By default, StatefulSets retain their storage. Kubernetes does not automatically delete the storage that was attached to a StatefulSet. The same can be achieved with deployments, but its not the default.