Although not widely reported, there are a lot of Muslims that don't think building the Mosque is a good idea. I've read a number of articles about this.
Arab TV Director: Alleged ‘Muslim Anger’ Over Ground Zero Mosque Protests is a ‘Fabrication’
Media claims that controversy is fomenting ‘Muslim extremism’ are not based in fact.
http://www.cultureandmediainstitute.org/articles/2010/20100901152241.aspxThe director of the popular Arab-language TV station Al Arabiya says that the Muslim world is not angry over increasing American opposition to a proposed mosque at Ground Zero, and that any claims to the contrary are attempts to “fabricate a conflict.”
“The lack of a unified stance throughout the Islamic world should be seen as response to the current attempt by some to 'fabricate' a conflict, claiming that Muslims are angry with the refusal to build a mosque in such a controversial setting,” wrote director Abd Al-Rahman Al-Rashid in an Aug. 29 column in a London daily. The column was translated and posted on the website for the Middle East Media Research Institute.
And the NPR reported on Aug. 24 that “Experts worry the controversy surrounding an Islamic center near ground zero in Lower Manhattan is playing right into the hands of radical extremists.”
But the director of Al-Arabiya TV – a competitor of Al Jazeera – said that the idea that the mosque controversy is inflaming anger across Muslim countries is nonsense.
...
He also said that many Muslims don’t want the mosque to be built, and understand why the project may be insensitive.
“[F]or many Muslims, building a mosque near the same land upon which three thousand people were killed by Muslims is not a necessity. Most comments from readers rejected the idea of building the mosque for fear of it turning into a symbol of hatred against Muslims,” wrote Al-Rashid.
And while the TV director said that the organizers of the mosque have “good intentions,” he also added that they have acted “without taking into account the serious nature of [constructing] a mosque at such a particularly sensitive time and place.”
This is not the first time Al-Rashid has spoken out over the Ground Zero mosque controversy. On Aug. 16 he wrote in another column that "Muslims never asked for" the proposed mosque at Ground Zero, and "do not care about its construction.”
“I can't imagine that Muslims [actually] want a mosque at this particular location, because it will become an arena for the promoters of hatred, and a monument to those who committed the crime,” he wrote. “Moreover, there are no practicing Muslims in the area who need a place to worship, because it is a commercial district. Is there anyone who is [really] eager [to build] this mosque?”