The only way that your intermediate shaft pulley will chew up your timing belt is if:
1) Your belt is running cock-eyed because the IP mounting bracket is bent, or/and your pump is not mounted properly.
and/or
2) Your belt is too tight. That can lead to a host of other problems like: worn intermediate shaft bearings, broken intermediate shaft, and eventual loss of oil pressure which can lead to a blown rod.
and/or
3) You intermediate shaft bearing is already worn/shaft is broken, and the shaft is riding in and out in the block. See #2 above
Even with a chip or two on the side of the pulley, it should NOT eat timing belts. If you can confirm that neither #1 no2 #2 above apply, then simply file down the rough edges of the pulley, and see what happens.