The decision to built this palace was taken by the Caffarelli, a rich Roman family, in 1500 and Raphael was asked to design it. One of his pupils, Lorenzo Lotto, supervised its construction.

The robust elegance of the palace was altered in the 18th and 19th century when the façade was enlarged and re-sited to face the Corso Vittorio Emanuele II and the top storey was added.

Tradition would have it that Charles V lodged here in 1536. in 1853 the Bishop of Perugia, the future Leo XIII, lived on the ground floor.

Inside some rooms are frescoed by Anton Raphael Mengs and by the school of Perin del Vaga. Before second world war the building became the seat of the Fascist party.

Today it houses ministerial offices.