Yep, Manifolds too.
Don't go mad, I generally just give stuff a quick fettle while the heads off.
Im no engine modifier by a long shot, just a general repair garage.
Not dealing with modifying heads daily I can't tell where the weak spots are, you gotta be careful porting heads as you can grind through into the water jacket or oil ways... not good!
Just take out rough edges, casting marks and blemishes and a quick tidy up is enough for me.
Another trick is to "waste" the valve guides to improve flow, some people grind the whole lot off that protrudes into the port, okay on a rarely used and regularly stripped race engine, not good for a daily drive!