Ella Jane Fitzgerald was born in Newport News, Virginia, the child of a common-law marriage between William and Temperance "Tempie" Fitzgerald. The pair separated soon after her birth and she and her mother moved to Yonkers, New York, with Tempie's boyfriend, Joseph Da Silva. As a child, Fitzgerald was placed in the Colored Orphan Asylum in Riverdale, the Bronx.

In her youth, she wanted to be a dancer, although she loved listening to jazz recordings by Louis Armstrong Bing Crosby and The Boswell Sisters. She idolized the lead singer Connee Boswell, later saying, "My mother brought home one of her records, and I fell in love with it....I tried so hard to sound just like her."

In 1932, her mother died from a heart attack. Following these traumas, Fitzgerald's grades dropped dramatically, and she frequently skipped school. At one point, she worked as a lookout at a bordello and also with a Mafia-affiliated numbers runner. After getting into trouble with the police, she was taken into custody and sent to a reform school. Eventually she escaped from the reformatory, and for a time was homeless.

She made her singing debut at 17 on November 21, 1934 at the Apollo Theater in Harlem, New York. She pulled in a weekly audience at the Apollo and she won the opportunity to compete in one of the earliest of its famous "Amateur Nights." She sang Hoagy Carmichael's "Judy" and "The Object of My Affection", a song recorded by the Boswell Sisters, and won the first prize of US$25.00.

In January 1935, Fitzgerald won the chance to perform for a week with the Tiny Bradshaw band at the Harlem Opera House. She met drummer and bandleader Chick Webb here for the first time. Webb offered her the opportunity to test with his band when they played a dance at Yale University. Despite the rough crowd, she was a great success, and Webb hired her to travel with the band for $12.50 a week.