I use a brass punch and give it a good smack, comes right off.
If you use a steel punch, the tool steel is harder than the pulley so it tends to deform the pulley but the brass is not as hard, so you can really whack it without cutting into the metal.