Conneff was born in 1945, in the heart of Dublin, in a district known as Liberties. Conneff's family loved music, especially legendary Irish tenor John McCormack and Caruso. Conneff was not exposed to traditional Irish music in the same way other members of The Chieftains were. He preferred pop music in his early years, but gradually came to appreciate the traditional music of his country. He began singing it and learned how to play the bodhrán to accompany his singing.

At one time Conneff and some friends ran the Tradition Club in Dublin where future bandmates Paddy Moloney and Seán Keane played. When Conneff was 31, he got the call from The Chieftains and has been a mainstay of the band ever since.