I found in the 1880 Census of Brooklyn another family that looks similar but is, i believe different; mixed up in some records
Charles F Baldwin b~1836, born in NY, father born NJ, mother born NY
wife: Mary E b~1837, born NY, father born Ire, mother born NJ
kids:
Annie F b~1863
Joseph M or N b~1865
Frederick W b~1865 (maybe twin with Joseph?)
Address: 19 Park Place, Kings, Brooklyn NY
