In the late Anglo-Saxon period North Elmham was a focal point for the Bishops of East Anglia. The bishopric was moved to Thetford in 1071.
Bishop Herbert de Losinga [ODNB] founded a church, after 1091, on the site of the earlier Anglo-Saxon cathedral. At some point after 1388 Bishop Henry le Despencer turned the former chapel into a castle. Part of the walls within the inner moat can be seen to the right of the chapel’s apse.
The chapel is now in the care of English Heritage.