[lvlrsSTI]

I believe Malay (polynesian) ancestor should be from Fujian-Zhejiang region and was displaced to Taiwan (where the aborigines have the purest Malay/Polynesian genes) when Han groups moved south.

But even if there were Malay ancestors who were originally in Hainan, they were probably displaced by the Li ethnic group many centuries ago.
The Li language is under the same family tree as Thai - e.g. The pronunciation for "Today" is similar to Thai "Wan ni" (video below).
I believe Hainanese also adopted some of the Li words as I just found out recently that the Laos pronunciation for chopsticks is very similar to Hainanese "Muey Du" !

You are right, before First Emperor conquered the southern China, there were hundreds of tribes of Yue people living there, dunt think they are related to Malay, they are more related to Thais and Vietnamese. They fled to further south to where they are today when Chinese took over Southern China. Having said that, Malay and Thais used to live in Yunnan and Guangxi provinces and migrated to the south over past 2000 years. Native people in Taiwan and Hainan are related to Malay, they tattoo their faces. 

There is an interesting history about Kingdom of Champa.

In southern part of today’s Vietnam there was a Kingdom of Champ founded by Malay people migrated from Brunei. 

After Vietnam conquered the kingdom in 1450, many got killed, some fled to Cambodia and some fled to Hainan Island, their descendants (a few thousands) are still there in Hainan today speaking their own language, Hainanese  and Mandarin…they have their own community/villages in Hainan. There are YouTube videos about them, people even talked to them in the street in Hainan.