homes were showered with asbestos when the Locarno burned.
Residents, oblivious to the possible dangers, have reported clearing up the dangerous building material with brooms and dustpans.
Swindon Council sent letters to homes in Beatty Court and Saxon Court the day after the blaze.
Asbestos cement is not considered dangerous unless it is picked up or otherwise disturbed.
Council workers have been on site clearing up the material on several occasions since Sunday night.
But residents feel more could have been done to keep them informed.
Saxon Court resident Shay Barry, 26, said: "They should have told us straight away. It's bad when you consider that children have been playing outside."
Flo Garrard, 68, who lives in Saxon Court, said: "I saw a man with a white suit and a mask picking it up with gloves on on Sunday night.
"I asked him what he was doing and he said he was picking up asbestos. It was falling all over the place when the fire was on."
In the council's letter, health and environment leader Steve Harcourt said: "If undisturbed, asbestos cement is completely safe as the asbestos fibres are locked in the cement and pose no threat to health. Even when damaged the majority of the fibres are still contained within the cement material."
He added: "I would emphasise that this material poses no threat to health if it left untouched."
He said it was unlikely there would be a risk to residents.
Anyone who finds what they think is a piece of asbestos should contact Swindon Services on 077860 20852.
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules hereComments are closed on this article