While reading Gerald of Wales Gemma Ecclesiastica (written about 1200), I came across this :
There is no dispensation [from the vow of celibacy], and there can be absolutely no recall of the vow except through a general council of the pope and cardinals with the consent of the whole Church. Such a council could permit single priests to marry, as is the practice among the Greeks. It is said that Alexander III had definitely proposed and firmly decreed to permit the marriage of priests because he was aware of the great dangers in the Western Church which stemmed from the vow of celibacy. [To this end] he had acquired the consent of the entire Roman Church, except for that of his abbot chancellor – a man of singular austerity who became Pope Gregory IV [sic – actually Gregory VIII], the third pope after Alexander III. There is nothing in either the Old or New Testament, in the Gospels, or in the writings of the Apostles, to prohibit marriage. The clergy of the Western Church have only been urged to celibacy by the holy fathers and apostolic men of the early Church for the sake of greater purity and integrity. The Eastern Church, however, adhered rather to the kind disposition of St. Paul (even though he himself had taken a vow of celibacy).
The reforming popes of the Middle Ages tried to get the secular clergy to live up to monastic standards of chastity and poverty, without much success. German bishops at the Council of Trent complained that almost all their priests were living in concubinage, and had purchased from the Roman Curia dispensations from episcopal attempts to enforce celibacy.
Trent and later reforms had more success, and set the parish priest apart from ordinary masculine life of sex, drinking, fighting, etc. This gave Catholic anti-clericalism a boost. Andalusia in Spain has an extreme version of this. Men do not go to church; women do. The men suspect the priests are seducing the women. If a priest is obviously chaste and circumspect in his dealings with women, men think he is a pederast. Priests can’t win.
Celibacy is not the only problem – Protestant ministers also are suspected of having designs upon the females under their care. But celibacy seems to exacerbate suspicions.