STEP BY STEP
For this process we’ll assume we’re using a DSLR attached to a telescope, shooting single (non-stacked) images. For best framing, a telescope focal length of 1,000mm or shorter is recommended. Placing M43 close to the middle of the long frame edge works well but make sure it’s got enough ‘space-border’ so the final image doesn’t look cramped.
Concentrate on the outer regions. Take test exposures to ensure you’re capturing the faint detail. Don’t worry if the overall image looks bright. If the core is looking white because of overexposure you’re probably on the right track! A simple way to correct such an image is to dim it using the levels mid-point slider, then tweak the brightness and contrast.
You should end up with one deep image with lots of faint stuff visible and another which only includes the core region. Open both images in a layer-based editor, the deeper result on top. Align them so the stars match. Draw a selection region, roughly tracing the border of the over-exposed core. Copy the selection to the clipboard.
Now focus as accurately as possible. The Trapezium cluster in the centre of M42 is ideal for this. Using Live View, the four brightest cluster stars should appear sharp and separated. Once done, set the camera ISO to a low or middle value. Low gives better tone and image quality, but if your tracking isn’t perfect, a middle value allows shorter exposures.
Capturing the inner Thrust region is more straightforward because it doesn’t require such a long exposure. A good technique is to bracket your exposures. Identify what you think is the optimal exposure then use incrementally shorter exposures, followed by incrementally longer ones. Now you’re covered if your original assessment was a bit off.
Paste the selection as a layer mask for the upper layer. In Photoshop, this means holding down the alt key while clicking the new layer mask button. Click the layer mask in the layers display and apply a fairly large Gaussian blur so the sharp edges disappear. Tweak each layer using Curves until the combined result looks natural to the eye.
Layer mask button