As your sanding the blanks it is likely you are also eroding away the bushes, so you end up with microscopic particles of steel transferring onto the blank & being ingrained into it as you continue to sand away.
so, try and keep off the bushes as much as possible, wipe down the blank & bushes at the very least with a cloth at every change of grit, change your bushes every now & again.