In 1104, the city of Jbeil fell under the control of the Crusaders with the support of the Genoese fleet. They renamed the city Gibelet and appointed members of the Embriaci family as its governors, granting them the title of "Counts of Gibelet".

They built a fortress, reusing Roman stones from the site. The castle, surrounded by a moat, was defended by four towers, placed at the corners and by a central keep.

A wooden drawbridge connected the castle to the city. The castle was still used as a barracks by the soldiers residing in the city after the Mamluk conquest in 1289. The Ottomans, who conquered Jbeil in 1516, built the stone bridge that has survived to this day, connecting the castle to the city.