Calves are one of the most genetic of the muscle groups, in the sense that some are born with large calves and others may always be smaller, no matter how hard they work at it.
That said, unless you’re recovering from an injury, I think you should work every muscle group. I don’t work my calves as often as other muscles because they are already proportionally strong, but I still work them regularly.