Certainly go. As you probably already know, Yedikule incorporates the former Golden Gate. The castle is full of the atmosphere of Palaiologan decline and of the conquest. There was nobody else there when I visited.
While you're there, you might also go and have a look at Stoudios (AKA Imrahor Camii), which is not far away from Yedikule. Stoudios was the centre of Byzantine monastic culture; for the non-Orthodox layman, perhaps it's most important contribution was to the field of music.