In ancient Sumeria, the sign of Capricorn was often associated with the planet Saturn and the Sumerian deity Enki.  But the full picture of Capricorn as we know it today comes from the ancient Greeks.  In Greek mythology, Cronus was a Titan, and the father of the gods.  
Just as he overthrew his own father, Uranos, Cronus knew he would be overthrown by one of his children.  So, as each of his and Rhea's children was born, he swallowed them whole.  Hestia, Demeter, Hades, Poseidon, and Hera, of them, were swallowed before they could grow up and threaten his rule. 
Rhea wanted children to play with, so when Zeus was born, she wrapped a rock in swaddling clothes and gave that to Cronus instead.  Zeus was hidden away in a cave and raised by foster parents.  There are different versions of the myth.  In one version, Zeus was raised by a goat-tending nymph named Amalthea or Adamanthea, and in another, by the goat herself (also named Amalthea), but the one constant throughout all of them is that Zeus was raised on goat's milk. 
When Zeus was old enough, he released the imprisoned brothers of Cronus--the Giants and the Cyclopes--who gave him power over thunder and lightning.  With their help, Zeus was able to overthrow Cronus and force him to spit out Zeus' brothers and sisters. 
The goat's horn later became the Horn of Plenty or Cornucopia.  In some versions, the goat's skin became the Golden Fleece that Jason and the Argonauts searched for. 
In thanks for tending him as a baby, Zeus placed Amalthea in the sky, as the constellation Capricornus, the Goat.