Honorable Robert Shirley

"There are doubts as to the identities of the first and second holders of the Castle. Dugdale, in "The Antiquities of Warwickshire," gives the first as Robert Marmion and the second as Robert his son. Palmer, in "The History of the Town and Castle of Tamworth," says that the first baron was Robert Marmion, lord of Fontenay in Normandy and that the second was Robert his son, after having considered whether Robert Marmion and Robert Dispensator or Royal Steward were one and the same person; however, in his later work "The Baronial Family of Marmion" he altered this, saying that Robert Dispensator was lord of the castle in the time of the Conqueror, but that he fell into disgrace with Henry I, and forfeited his estates, which were then given to a kinsman, Sir Roger Marmion, who was the son of Sir Robert Marmion, Lord of Fontenay. This question has also been dealt with by J. H. Round in "Feudal England" (1895) in which he agrees with Palmer's later work, and says that the actual relation of the Marmions is a problem as yet unsolved, but that it is certain Tamworth Castle originally belonged to Robert Dispensator and equally certain that it was held successively by Roger Marmion and his son Robert under Henry I.
The date of the building of the castle by its first Norman lord is not known, but was probably between 1070, when William I had completed his conquests, and 1080.
The Marmions did not reside at the castle continuously. They had other possessions, including, probably from the time of Sir Roger Marmion, the manor at Middleton, a few miles from Tamworth, and when they were not engaged in wars abroad or at home, they lived sometimes at the Castle and sometimes at Middleton.
The way in which the castle descended from the first lord to its acquisition by Tamworth Corporation in 1897 is summarized below:

1688: Robert Shirley, by his marriage to Anne Ferrers."
[http://www.tamworthcastlefriends.org/page9.html#:~:text=1554%3A%20J....]