Millom castle was built by Godard de Boyvil who was in possession of the first manor of Millom. The Lordship of Millom at this time, was such a senior position, that it had complete jurisdictional independence, meaning that sheriff of the county could not enter the castle uninvited!
The castle later passed to the Huddleston family after the last de Boyvil died with no male heir. The castle saw action during the English Civil War, and was badly damaged in a one canon attack in 1648.