We report on the observation of a deficit in the cosmic ray flux from the directions of the Moon and Sun with five years of data taken by the IceCube Neutrino Observatory. Between 2010 May and 2011 May the IceCube detector operated with 79 strings deployed in the glacial ice at the South Pole, and with 86 strings between 2011 May and 2015 May. A binned analysis is used to measure the relative deficit and significance of the cosmic ray shadows. Both the cosmic ray Moon and Sun shadows are detected with high statistical significance (>10σ) for each year. The results for the Moon shadow are consistent with previous analyses and verify the stability of the IceCube detector over time. This work represents the first observation of the Sun shadow with the IceCube detector. We show that the cosmic ray shadow of the Sun varies with time. These results make it possible to study cosmic ray transport near the Sun with future data from IceCube.