The yellow in the middle of the picture is Max II castle, Schloss Hohenschwangau. Ludvig II spent his childhood here. It was also here Ludvig II met Richard Wagner who gave him the inspiration to Schloss Neuschwanstein.
Castle Hohenschwangau was built in the 12th century but was renovated by Max II in the 1830's.
The work on the castle began in 1869 on orders from Ludvid II. It was never finnished. When Ludvig II by "mysterious circumstances" died in 1886 the progress on the castle stopped.