On 18 April 1785, he had married Lady Hannah Maitland (a daughter of the 7th Earl of Lauderdale)

As a result of the marquess's declining health, he and his wife went to travel the Continent in 1802, starting in France. It was here that they were captured by Napoleon's police a year later, with other British subjects, when war was renewed between the two countries. They were then imprisoned in the fortress at Verdun and the marchioness died there on 8 May 1804, as did the marquess in the following August.