You can port the exhaust, but the problem you may run into is making the ports on the head bigger than the ports on the exhaust manifold, which would be a big problem. It's actually a little better on exhaust to have a "step" where the manifold is a tad bigger than the exhaust port to prevent pressure reversion. But yes, I'm in the camp that the exhaust side (porting, and manifold) needs opened up much more, but to really take advantage of that, some manifold work and/or custom manifolds are needed and that is out of the realm of most of the people that want a slightly peppier car without a ton of effort.
Also, with exhaust tubing diameter, there is a popular belief that you need backpressure to make power and that by removing backpressure it kills low end torque. This is so completely untrue- in a naturally aspirated car, there is a point in exhaust size where maximum gas velocity occurs- this is the point at maximum efficiency of gas flow which helps to scavenge the cylinders. Going to a larger exhaust after that point will only slow down velocity which will hurt engine efficiency. Backpressure is NEVER a good thing and by increasing exhaust velocity you are actually eliminating as much backpressure as possible.
Turbo cars are different. With a turbo, the restriction is the turbine itself- there is always a lot of backpressure in the exhaust manifold before the turbo- hence the main downfall of a turbocharged system. You want the exhaust gasses to escape as fast as possible to keep stress off of the turbine and to encourage it to spool- therefore the general rule with a turbo is to make the exhaust as big as you can fit- exhaust gas velocity makes much less of a difference post-turbo, since there is no cylinder scavenging going on, but you can encourage exhaust velocity by making a tapered "cone" exiting the turbine; i.e. say your turbine outlet is 2" and you have a 2" pipe that gradually expands to a 3" pipe over a certain distance. This has proven to increase turbine efficiency a small degree. Theoretically, the best turbo exhaust system is the aforementioned "cone" out of the turbine that dumps directly out after that. Sometimes you will see the NHRA sport-compact drag cars with systems very similar to that.
HTH,
Brendan