[Ali Imran]

All destroyed lah...

Not really. Once a place has been established as a temple, people would always use it to worship God, even when the structure is no longer there. 

When the Caliph Umar conquered Jerusalem, he was shocked that the Christians were using it as a dumping ground. The first thing he did was to clean the temple with his bare hands.