I would scrape them off without using much force. Otherwise they can come loose on a worst time possible and maybe short circuit some pins when the board is powered. Probably will never happen if the raspberry is statically mounted sonewhere, but you never know
P.S If you are going to run some servers/services off the raspberry do not use regular usb sticks or sd cards. Get a USB ssd or USB to sata or M2 adapter. Some write intensive applications burn through quality sd cards/usb sticks within a year