Red doesn’t always convey hatred towards barbarians, sometimes it is just symbolising the hatred of other ethnicities in general. Also yellow doesn’t always convey literal gold, it can also be territories (because the land can then be worked to make money). Examples:

NOTE: While there were widely-recognized iconographies for representations of imperial power (the eagle, Jupiter’s thunderbolt, SPQR), there was no standardized flag. as we would recognize the concept, for the Roman Empire.