The House of Grammont is mentioned as early as the 14th century as a family of power. The founder of the main house was Hughes II Granges and his descendants divided the House into several branches: Grammont-Fallon, Châtillon, Conflandey, Mélisey, Vellechevreux and Vezet.
The family belonged to the region of Franché-Comte. The original château they resided in was ordered demolished by Louis XIV after his taking of Besancon; consequently, the family built a new castle, the Château de Villersexel. In 1718 the Lordship of Villersexel was raised to a Marquisate for Michel-Dorothée de Grammont who served as Lieutenant-General in the royal army.
Théodule de Grammont married a daughter of the Noailles-family which was considered an advantageous match since the House of Noailles was one of the greatest in France.
The family continued to reside at Villersexel till the revolution in 1789.