He developed precociously as a musician. At the age of thirteen he composed and conducted a cantata for chorus and orchestra based on selections of Ovid's Metamorphoses. He was also a fast-developing pianist and violinist. He studied at Cologne Conservatory with Franz Bölsche in music theory, Lazzaro Uzielli in piano, and Hermann Abendroth in conducting. He won the Wüllner prize in conducting in his graduation year of 1920.

He obtained a position conducting at Cologne Opera, where he was assistant to Otto Klemperer. When Klemperer left in 1924, Steinberg received the appointment as Principal Conductor. In 1925 he accepted the post of conductor of the German Theater in Prague. In 1929 he became musical director of the Frankfort Opera. His tenure there was marked by an interest in modern opera. His productions included Berg's Wozzeck, Schoenberg's Von heute auf Morgen, Antheil's Transatlantic, and Weill's Aufstieg und Fall der Stadt Mahagonny.

The advent of Nazi rule in 1933 ended his career. He was restricted to conducting concerts for the Jewish Culture League in Frankfurt and Berlin. This was an insidious creation of the Nazis that both furthered its institutionalized anti-Semitism by creating a segregated organization for a segregated orchestra, while preserving the illusion that the Nazis goals went no further than ethnic separation. Steinberg left Germany in 1936 for Palestine, where he conducted the new orchestra there that eventually became the Israel Philharmonic. The Palestine Philharmonic's first concert was conducted by Arturo Toscanini. After working with Steinberg, Toscanini invited him to go to the United States as associate conductor of his NBC Symphony Orchestra. Steinberg took up that position in 1938.

Toscanini and Klemperer were Steinberg's two mentors. He adopted their clear, faithful approach to the classic scores and, like Klemperer, lost much of his early interest in Modern music. Steinberg guest conducted regularly during his tenure with the NBC SO. In 1945 he became Music Director of the Buffalo (New York) Philharmonic Orchestra. In 1952 he obtained the major appointment of his career, as music director of the Pittsburgh Symphony Orchestra. He restored that orchestra to an artistic high point. Concurrently, he was musical director of the London Philharmonic Orchestra (1958-1960).

In 1960 he scored a great success guest conducting the Boston Symphony Orchestra and was the preferred choice of its board for their next music director, as Charles Munch was stepping down from the position. However, RCA, the orchestra's record company, successfully pressured them to appoint Erich Leinsdorf, already on their roster of conductors. After Leinsdorf's tenure, one of mixed success, ended, they did appoint Steinberg to the post effective 1969. This was also only a partial success, because then health problems interfered with his abilities and caused frequent substitutions. He left the position in 1972 and restricted his activities.