The Eastern Orthodox Church, which names Joseph's first wife as Salome, holds that Joseph was a widower and merely betrothed, but never married, to Mary, and that references to Jesus' "brothers" are to children of Joseph and Salome. The position of the Catholic Church, derived from the writings of Saint Jerome, is that Joseph was the husband of Mary, but that references to Jesus' "brothers" should be understood to mean cousins or step-brothers. In both cases, the church doctrine of the Perpetual Virginity means that Joseph and Mary never had sexual relations. The Protestant churches, following the tenet of Virgin Birth but not that of Perpetual Virginity, hold no strong views on the subject.
Source: Wikipedia, Saint Joseph