Nathan Sidwell said in a post directed to the committee that if it was unwilling to remove Stallman, then the panel should explain why it was not able to do so.
Stallman is also the founder of the GNU Project and the original author of GCC.
"RMS [Stallman] is no longer a developer of GCC, the most recent commit I can find regards SCO in 2003," Sidwell wrote in a long email.
"When was RMS’ most recent positive input to the GCC project? Even if it was recent and significant, that doesn’t mean his toxic behaviour should be accepted."
Sidwell's post has mostly been supported by other GCC developers, but there has been the odd note of dissent.
A push to oust Stallman from the FSF board began last week soon after he announced that he was rejoining the board.
Individuals involved in free and open source software have published a letter calling on Stallman to go, while his supporters have penned their own letter, opposing his ouster. The pro-Stallman group has mustered 3632 individual supporters, while the anti-Stallman letter has 16 main signatories, 48 companies and organisations, and 2811 individual supporters as of this morning.
Stallman announced he was rejoining the board during the FSF's annual LibrePlanet conference on 19 March.
Both Red Hat and SUSE, the two largest open source vendors, have supported the move. But while SUSE has put its name as a company to the letter calling for Stallman to be removed, Red Hat has not done so.
Meanwhile, the Open Source Initiative, an organisation that is involved in community-building, education, and public advocacy to promote awareness and the importance of non-proprietary software, also called for Stallman to be thrown out.
In a statement issued last week, the OSI said: "The Open Source Initiative calls upon the Free Software Foundation to hold Stallman responsible for past behaviour, remove him from the organisation’s leadership and work to address the harm he caused to all those he has excluded: those he considers less worthy, and those he has hurt with his words and actions.
"We will not participate in any events that include Richard M. Stallman and we cannot collaborate with the Free Software Foundation until Stallman is removed from the organisation’s leadership."
Well-known security practitioner Katie Moussouris, the chief executive and founder of Luta Security, was one of the first to call out Stallman last Tuesday.