Captain Henry Brown built Cedar Grove for his daughter Nancy Ann Steptoe. She died before the house was completed in 1819. Her husband was Dr William Steptoe and her son, Edward Jenner Steptoe rose to the rank of Lt. Col in the Army during the Mexican American War. Dr Steptoe remarried and lived at Cedar Grove until his death in 1862.
The Yancey family acquired the home some time in the late 1800s. Rebecca Yancey Williams recounted her family history including descriptions of events that occurred at Cedar Grove in her book "The Vanishing Virginian". The 1941 MGM film of the same name was based on Mrs. William's book.