Today this term simply means “never” and is used in a number of different circumstances. This line is particularly used by women when asked out by men they do not fancy. Despite its more mundane meaning today this phrase actually has a military origin.
During the First World War Prussian military officers believed in Der Tag (The Day), when the German military would rise to prominence and replace the British as the most important in Europe. Der Tag was a common toast of German officers and it was often the theme of many newspaper articles and books. The phrase became so popular that the British military countered it with “that’ll be the day.” British soldiers also used this phrase to yell across no-man’s land to taunt the Germans. Eventually this phrase became so popular that it was used as the title for songs, books and films.
One popular version of this phrase comes from the song That'll be the Day by Buddy Holly in 1957.
For all the men that are told by women “that’ll be the day” just remember that they are referring to the inconceivability of German military dominance and not any inadequacies that you may have.