On Billy goat means: If you are really religious, then Billy goat means that you cannot say anything on God. You can just use billy goat to express your faith. (in Community Dictionary, added by Karla Frost)