Theres a couple of things to consider:
If you are using an intercooler, then sure, its better than nothing to try and insulate the intake after the IC, to keep the air cool on its way to the cylinder.
If, however, you aren't intercooled, then its wont really make any difference. The reason the turbocharger compressor housing is hot, is 99% due to the fact that its compressing air and the air gets hot, which then heats up the compressor housing. Likewise, the hot air is going to heat up the intake manifold, so isolating it from the compressor is pretty pointless.
At the same time, yes, the exhaust side is right there aswell, and it is much hotter. Anything you can do there will be a benefit. On my old AAZ, I made up a stainless steel sheetmetal heat sheild, and just bolted it to the turbocharger exhaust side (using one of the downpipe studs to mount it) which seemed to help stop heat soak into the manifold (like in traffic).
But, the main over-riding principle here is that because the mass flow rate of the intake air is so large - all these things only make a very small difference - because, the most significant factor attributing to the temperature of any of the intake parts, is the temperature of the air flowing through it.