When the small seal, which surrounds the input shaft of the fuel pump wears (front pump seal), starting becomes annoying just like you have stated. However, usually it is more of a white smoke if memory serves me.
This is one way the pump loses the prime. And this will just drive you nuts....plus you will learn to rebuild your own starter!!!!
A specific seal puller makes the job a breeze. Otherwise, you've go to use sheet metal screws, etc. and make sure NOT to damage the shaft or housing of the pump. If you still have the original seal.....man, they come out hard!!!!!!!