Smith was born in Bell, California, and began playing the bass at the age of eight, inspired by his older brother, jazz musician Carson Smith. He made his performing debut aged 13 at the Compton Community Center.

He went on to perform with Thelonious Monk, Art Blakey, Duke Ellington, Billy Eckstine, Diane Schuur, Lee Konitz, Bruce Forman, Jackie and Roy, Carmen McRae, Gary Foster, Art Farmer, Alan Broadbent, Mose Alison, Bob Brookmeyer, Blue Mitchell, Erroll Garner, Art Pepper, Mason Williams, Percy Faith, Burt Bacharach, Ray Charles, The Manhattan Transfer, and Johnny Mathis. He also works as a session musician, and has played on recordings by Sonny and Cher, The Beach Boys, and The Righteous Brothers, among many others.

Smith was playing with Monk at the Los Angeles jazz club Shelly's Manne-Hole when he was spotted by director Guy Hamilton, who cast him as the assassin "Mr. Kidd" (alongside Bruce Glover as "Mr. Wint") in the 1971 James Bond film Diamonds Are Forever. He went to have several other minor acting roles on film and television.