All this would change in 1964 when his band, the Soul Giants, auditioned a young man named Frank Zappa for lead guitarist. Zappa could smell change in the wind big time and eventually suggested the band quit doing covers and let him write some songs specifically for them. The idea was to create an image of the band as the ultimate set of freaks, to cash in on the peace-and-love hippie era while simultaneously making fun of it, and everything else that came to mind. Zappa also envisioned himself as an avant-garde composer creating works along the lines of Edgar Varese. It was all a tall order, but with the band's name changed to the Mothers of Invention and the 1965 release of the first double album in rock history, Zappa's twisted dream became a reality.

Black went from backing up bumping and grinding to playing at Royal Festival Hall in a few years, but was dumped back out on the street when Zappa decided to disband his original group of Mothers and embark on a solo career in 1970. By then Zappa and the band had released their first and only full-length commercial film, 200 Motels, in which Black was prominently featured in what came to be one of his signature songs, "Lonesome Cowboy Burt." Black and several other members of the Mothers rebounded with a new band, Geronimo Black. The group's direction was less of a freakout and more rock and blues, and Black wrote several classic Native American protest tunes for the band's debut album. Black and the group were based out of California up until 1973. A shift in record company management led to the band being dumped, beginning several decades of struggle for Black as he moved around the Western United States, almost always having to work at some job other than music in order to survive and support his five children.

For Black, being impoverished wasn't just a private matter. His former boss, Zappa, had recorded band meetings and arguments, inevitably about money and the lack of it, and had edited these spoken word bits into several Mothers releases. This material had an enlightening effect on many young listeners who thought all rock bandmembers were rolling in amassed wealth and hadn't a care in the world, certainly not the worries that plagued middle-class adults raising families. But no matter how hard times became, Black always remained involved in some kind of musical project. From 1973-1982 Black lived in New Mexico and Texas. He performed in a group called the Lotus Family with the banjo player Jim Bowie. Other Black groups from this period were Captain Glasspack and the Magic Mufflers and Big Sonny and the Lo Boys. In 1975, Black joined the band of another avant-garde rock legend, Captain Beefheart, as one of a set of double drummers. He stayed with the group for less than a year, performing on the Chicago-based Soundstage television program and at the Nepworth Pop Festival in the United Kingdom. He was asked to do some guest vocals on a Zappa album in 1981, producing the off-color country song "Harder Than Your Husband." But this friendly association with Zappa ended when most of the original Mothers decided to sue for unpaid back royalties, winning the case but not allowed to discuss the outcome with the press.

In 1982 he moved to Austin, TX, a town with a promising music scene that led to some new relationships, including one with the British singer Arthur Brown, another individual attempting to survive on a few moments of '60s fame, in his case a bizarre hit single entitled "Fire." That Black and Brown would eventually combine their talents as house painters as well as musically says more about the lack of sufficiently paid musical employment in the Austin area than it does about these men's versatility. Throughout this period there had been attempts to revive the original Mothers, sans Zappa, as the Grandmothers. Most frequent partners in these ventures would be keyboardist Don Preston and reed player Bunk Gardner. While living in Austin, Black also began a Texas-based version of the group, and began to incorporate not just senior ex-Mothers but young Zappa fanatics eager to take part in a repertory company of Zappa music alongside their hero's former sidemen.

In 1991, Black and Preston were both invited to perform at the Moers Jazz Festival in a 12-piece grouping of musicians from different genres organized by guitarist Eugene Chadbourne. One year later Black had the opportunity to relocate in Europe when his wife was offered a job teaching on a military base in Vicenza, Italy. The Army wasn't particularly fond of the Blacks and wanted to send them back to Texas, but they opted to stay in Europe, relocating to the Stuttgart area of Germany where, in collaboration with the Muffin record company, he began an intensive effort to expand work possibilities for the Grandmothers. At the same time, Black continued his relationship with Chadbourne. The two formed a duo named the Jack and Jim Show after a painting Beefheart had done of Black with a jackrabbit. Black began getting enough work in Europe to survive as a musician, backing up a variety of blues singers in a band led by guitarist and harmonica player Rick Farrell and also appearing from 1994 onward as a guest vocalist with the British band the Muffin Men.

Concentrating more on singing than drumming, Black became known as one of the best interpreters of classic Zappa and Beefheart material. Despite threats of legal action from Zappa's widow, Gail, the Grandmothers continued to increase their activity. In 2000 the band embarked on a two-month tour of the United States, playing 60 shows in 66 days. Black continued in the life of a journeyman musician. "I'm famous, but I don't have a pot to piss in," he was fond of saying. The University of Liverpool Press planned to publish a biography of Black in 2002, to be written by Black and Rod Gilliard of the Muffin Men. Jimmy Carl Black passed away November 1, 2008, after a bout with cancer.