We address the problem of amplify-and-forward (AF) relaying in a hybrid satellite-terrestrial communication system with unknown channel gains. A masked destination earth station (ES) receives the signal forwarded by a relaying ES, i.e., from a terrestrial link. There is no direct transmission from the satellite to the destination ES. The satellite-relay ES (satellite link) and relay ES-destination ES (terrestrial link) links are assumed to follow shadowed Rician fading and Nakagami-m fading, respectively. A scheme for channel estimation and detection of transmitted signal from the satellite to the destination ES is proposed. The terrestrial link is estimated separately, whereas the cooperative hybrid link (satellite-relay-destination) is estimated jointly with symbol detection. The average symbol error rates of the M-ary phase-shift keying and M-ary quadrature amplitude modulation constellations are derived for the considered hybrid communication system. Moreover, we also derive the analytical diversity order of the considered system with estimated channel gains. It is shown by simulation and analysis that the estimation error does not affect the diversity order of the system. Furthermore, the average and asymptotic capacity of the system is also derived by using the moment-generating function approach.