Longo came from a musical background. His father played bass, his mother played organ at church, and his music training began at a young age. Mike recalled seeing Sugar Child Robinson playing boogie woogie piano: "The first time I saw him, man, he knocked me out. I must have been 3 or 4 years old. He played after the Count Basie show, so I went home and started picking out boogie woogie bass lines." His parents took him for formal lessons at the Cincinnati Conservatory of Music at four. He moved to Fort Lauderdale, Florida soon after. At the age of twelve he won a local talent contest.
Mike's career started in his father's band, but later Julian Cannonball Adderley helped him get gigs of his own. Their working relationship began before Adderly was well-known as a band leader. Adderley approached the teenaged Longo because he needed a pianist at his church. At this time the town was largely segregated so the white Mike Longo playing at a black church was somewhat unusual. When this led to recordings with Adderly in the mid-1950s, Longo was initially too young to go to the clubs with him. Still in 10th grade, one of the places Longo played was Porky's, which was later portrayed in the movie. Mike would go on to receive his Bachelor of Music degree from Western Kentucky University.
The great Dizzy Gillespie first heard Longo when he was playing at The Metropole. "I was playing downstairs with Red Allen, and Dizzy was playing upstairs with his band. So every time he wanted to go outside for a break, he had to come down the stairs and pass us on the way out. There was a joint across the street called the Copper Rail, which was a soul food restaurant and a bar where the musicians from the Metropole would all hang out. Soon I learned Dizzy mentioned me in an interview in International Musician, the musician unions magazine, when he was asked about any promising young musicians he heard". He was a fan of Oscar Peterson from a young age and he studied with the master pianist from 1961-1962. "In terms of technique, touch, I was playing with locked wrists and too much arm technique. The main thing I got from Peterson was how to play piano and how to be a jazz pianist- textures, voicings, touch, time, conception, tone on the instrument."
In the 1960s Longo began to lead the Mike Longo Trio, which would remain active for the next 42 years. Gillespie was playing at the Metropole again and Longo, depressed after divorcing his first wife, had just gotten a gig at Embers West playing with Roy Eldridge. Eldridge brought Gillespie to see Longo playing with Paul Chambers. Gillespie hired Longo the next day. Longo eventually became musical director for the Dizzy Gillespie Quintet and later Gillespie chose him to be the pianist for the Dizzy Gillespie All-Star Band. From 1966 onward his music career would be strongly linked to that of Gillespie's. In 1993 he delivered a eulogy at his funeral. He was also with Gillespie on the night he died.
Longo performs at weekly jazz sessions held at the New York City Bahá'í center in honor of Gillespie, a tradition he helped start. Like Gillespie he is a member of the Bahá'í Faith. Longo also teaches a master class to upcoming jazz musicians including Adam Rafferty and John Austria. Longo's big band, the New York State of the Art Jazz Ensemble, plays at the center intermittently and provides upcoming musicians a chance to learn on stage and the audience receives a jazz experience at reasonable cost. A big part of Mike Longo's mission is to reestablish the apprenticeship relationship in teaching jazz. He says "I know jazz education is an important thing and I know that the field means well, but there seems to be a trend in that field to teach jazz where people are actually copying off recordings instead of actually learning to play jazz. The apprenticeship aspect of jazz has always been the way it has evolved."