How to Use selections and masks in Photoshop
Path to success
There’s more than one way to skin this particular cat, but we’ll start with the Pen tool, which pros favor for sharp cutouts. We traced around our furry subject — the path doesn’t need to be perfect, just a general outline.
Convert to selection
In the Path window, Ctrl-click your path and choose Make Selection. Keep Anti-aliased checked, but make sure the feather radius is set to zero. Click OK and the familiar marching ants marquee appears where your path was.
Select and Mask
Press Cmd+Opt+R to open the Select and Mask workspace. The first slider in the panel is Transparency. Drag this to the right to preview how your cutout looks. Right now it’s pretty rough and ready, with no fine fur detail.
So refined
Press R to select the Refine Edge tool, tap the left or right square bracket key to get the brush a good size to work with, and paint around the outline. Photoshop will detect the fine edge detail and add it to the selection.
Paint away
Here’s the Refine Edge tool at work — as we painted around the monkey’s previously roughly cut head, Photoshop has automatically detected and added tons of fur detail. Work around the rest of the outline of your image.
Going manual
Here, Refine Edge has failed us and left a bit of our infant monkey’s head transparent. Fix things like this manually — press B to switch to the Selection Brush, then paint the selection area back in. Again, a soft brush works best.
Smear attack
The top of our monkey’s head has a “halo” of the old background around it, which will look like a smear. Select the Brush tool (B), hold Opt so your brush will remove areas from the selection, and paint out all the smears.
Corrections
Using a soft brush, work in small areas, and check the results before moving on. Step back by pressing Cmd+Opt+Z, and redo strokes with Cmd+Shift+Z. Zoom in (press Cmd and +) and vary brush size to get fine detail right.
Check your work
Press K to view an outline of the selection on a plain black background. You should see plenty of fine detail and no hard edges. If you see any of the latter, go back over the area with the Refine Edge brush to fix it.
Fine-tune and export
Use progressively smaller brushes to refine your work, then decide how you want to export the cutout in Output Settings. Selection (marching ants) is the most basic; a new layer with a layer mask is the most flexible.