Sir Kenneth had a 'servant' called James Fraser, as shown on the marriage records for James Fraser and Mary McKenzie (married on 29th July 1807 in Gairloch parish) and the baptismal records for their daughters Isobell (b. 1810) and Jean (b. 1812). A transcript of the marriage banns which I have seen lists Sir Kenneth as James Fraser's 'father'. I think that James Fraser worked as Sir Kenneth's 'farm grieve', although I can't source that fact.
I am interested in this question regarding the relationship between Sir Kenneth and James Fraser because I had a pair of great great great grandparents called James Fraser and Mary M(a)ckenzie of the same generation as the couple at Kerrysdale. That James Fraser was known to have been a farmer, but thought to have had roots in Kiltarlity. However, there is a family story about his daughter, my great great grandmother, known as Ann Fraser (b. circa 1810) being invited to visit the house of some local Mackenzie landowners (in the vicinity of Dingwall, where the Mackenzies of Gairloch also lived, at Conan Mains) in the 1850s after she was widowed, apparently in order to be given assistance as a family member of some sort. Hence, I wonder if her father could have been a natural son of Sir Kenneth.
She was - or so the story goes - shown a picture of her mother, who was referred to by a 'Francis' Mackenzie (described as a bachelor) as his 'aunt'. There was a Francis Harford Mackenzie of Kerrysdale who would have been about 20 at the time, who was the brother of Sir Kenneth Mackenzie, the 6th baronet of Gairloch, and the Mary Mckenzie at Kerrysdale would actually have been the wife of his first cousin once removed if James Fraser had been his great uncle's son.
Meanwhile, Sir Kenneth would still have been alive then and living at Kerrysdale until his death in 1860 at the age of about 95, so Francis might have been entrusted with this errand on his behalf at Conan Mains.
The famously inadequate state of the parish records for Kilmorack and Kiltarlity, and probably Gairloch too makes it difficult to be very sure, but the marriage record of 1807 seems to be the only one in Scotland for the time discoverable for a James Fraser and a Mary Mckenzie. On the other hand, there is no record of the baptism of my great great grandmother, Ann, to parents of those names although her approximate date of birth makes me wonder if she could not have been baptised 'Isobell' or 'Jean'.
It is amazing how we can convince ourselves of the truth of our own pet theories about our ancestry based on configurations of facts and circumstances such as those in the no-doubt plausible network laid out here, and I dare say there are some other awkward facts in the family history that would run athwart them. Still, it would be good to hear any corroborating evidence regarding the possibility of Sir Kenneth having had a natural son called James Fraser although it would remain to be seen if that was the same man as my ancestor.