Mary Fraser, co-heiress of Oliver Castle

Mary was the wife of Sir Gilbert de la Haye of Locherworth and not of Sir Gilbert Haye, 5th of Erroll.
She was daughter and coheir of Sir Simon Fraser of Oliver Castle, who was executed by Edward I. 1306.

Born in Oliver Castle

Oliver Castle was a small tower house, located in the upper Tweed Valley in the Scottish Borders, within the bounds of the village of Tweedsmuir. The castle was originally part of the line of peel towers along the Tweed Valley. It was replaced in the seventeenth century by a house on the same site, and in the 18th century by a house on lower ground. For most of its existence the property has been owned by members of the Tweedie family.

The castle was originally associated with Clan Fraser, and was perhaps named for Oliver Fraser, whose gave lands to Newbattle Abbey as recorded in its register. The lines of descent from Oliver and his nephew Adam are uncertain, but the Frasers continued to exert power from Oliver Castle with Sir Bernard Fraser and Sir Gilbert Fraser, who held in turn the hereditary office of Sheriff of Tweeddale. A descendant, Sir Simon Fraser of Oliver and Neidpath, Knight Banneret, fought in the Wars of Scottish Independence. Clan Tweedie was a sept of Clan Fraser and by this time the Tweedies had already taken over the Fraser title to Drumelzier. In the 14th century, William Tweedie, a younger son of James and Katherine Tweedie of Drumelzier, obtained the lands of Oliver from the preceptor of Torphichen.

The legendary lawlessness of the Scottish Marches revolved around bitter inter-family feuds and conflict over livestock. In an incident characteristic of the times, Thomas Porteus of the adjacent Hawkshaw was arraigned on the 16th February 1489 for having lifted seventy-four lambs from the lands of Oliver Castle, belonging to William Tweedie and Lawrence Tweedie

The Castle of Oliver was on very high ground, 1,000 feet (300 m) above sea level, and by the late 1640s was in a ruinous state and past all repair. Patrick Tweedie decided to replace it, building a new house at Oliver in 1649, when the family moved from Oliver Castle itself. Later on, about 1731 or 1734, another house appears to have been built at or near the site of the present house of Oliver, and in 1786, Thomas Tweedie of Oliver built the current house there.