Pemberton House was built in 1754 for Pemberton Milnes 1729-1795, cloth merchant, active magistrate and leader of the dissenting Whig political interest in the West Riding