Close to Kilgarvan the remains of Ardtully House can be found. Although called a castle, Ardtully is an old 27 room mansion built in 1847, on the site of the ruins of an older castle. In 1921, during the Irish War of Independence, the IRA burnt the house down. You can find some more info on the history of this site here.
Though you cannot enter the remains of the house, the Ardtully estate is a cool place for a picknick or a walk and plenty of interesting things can be seen, such as an old kiln and the bridge over the Roughty river.
The Roughty River itself is another good reason to visit this place. I used to come here all the time for a swim, as the river downstream is just perfect for it. Upstream the Ardtully salmon fishery must make for good fishing.