http://userpages.monmouth.com/~mcha/ahhist.htm

The Allen House property was first built on between 1670 and 1688 by Quaker Judah Allen. This original structure, however, is no longer standing, and its location is unknown. In the early 1700s, the property passed into the hands of New York City merchant Richard Stillwell, who built the main part of the still standing structure ca. 1740 to be his second home. Richard Stillwell died in 1743, leaving the property to his wife and children, who tried several times in the early 1750s to sell the property.