Basically, "rubbing out" means the same thing as "sanding the finish". Like every other aspect of finishing, it should be practiced on scrap first.
After you apply 3 or 4 coats of shellac, give it 2-3 days to cure. Then lightly sand it, dry, with 400 or 600 grit paper. Stay away from edges and corners, or you'll cut through the finish. It will look terrible at this point, all white and scratched, but the idea is to remove dust nibs and brush marks, and to make it level. When a sheet of paper gets loaded up, change it. Try to produce a uniform scratch pattern, sanding with the grain. After this step, don't worry about grain direction at all.
After this initial dry sanding, all of the next steps are done wet, using a lubricant. You can use mineral oil, wax, mineral spirits, or soapy water. I prefer the oil, because it slows down the cutting action. If you use wax, remember one thing. If you should cut through the finish on a corner or edge, you'll have to strip the wax with mineral spirits before you go back and touch up the spot with more shellac.
You can continue to use finer and finer grits of wet/dry SC sandpaper, or you can use finer and finer steel wool, or you can use pumice and rottenstone. In any case, the idea is the same -- work your way up through the grits until you have the gloss level that you want. The other advantage of rubbing out -- a silky smooth feel to the finish -- will happen automatically.
If you choose sandpaper, you might use 600 grit, 1000 grit, 1500 grit, then 2000 grit, all with mineral oil. Be careful though, it's easy to cut through with sandpaper. Back the paper with foam. You could use a power-sander, but it's easy to cut through, so I do it by hand, just to be safe. I'll use a power sander for wide surfaces, like a tabletop, and just do the edges by hand.
With steel wool, you might use 000 followed by 0000, and use pastewax for the lubricant. I do this when I want to save time, or I want a more satin finish, and the finish is in pretty good shape to begin with. This is the easiest, shortest route.
The ultimate, time-honored method is to use pumice & rottenstone. I use mineral oil for the two grades of pumice -- 2F and 4F. The rating is similar to steel wool, where 2F is coarse, and 4F is fine. Basically, you just wet the wood, then sprinkle the powder, and rub. You use a pad, which is available from the same places that sell pumice & rottenstone -- woodworking specialty houses or mail order. You rub in a circular motion, working methodically across the wood. Wipe all of the 2F off before starting with the 4F. Then I clean the mineral oil off with mineral spirits or naptha. Then I use soapy water -- dish soap or Murphy's oil soap -- and the rottenstone. BTW, I keep two pads on hand -- one for the pumice, and a separate one for the rottenstone. After thoroughly rubbing with rottenstone, rinse and wipe dry, removing every trace of powder. The finish should look smooth and level, but it may have a white cast to it at this point. Don't worry about that.
The last step is to wax it. I use a fine abrasive pad, usually white or gray in color, and pastewax, usually Johnson's or Trewax, which you can find in the floor-care section of your hardware store. Thin the wax with a little naptha or mineral spirits, and rub like crazy. Wait 5 minutes until it hazes, then buff off. Come back in an hour, and buff again. This second buffing is an important step whenever you wax wood.
The finish will feel like silk, you'll be able to see yourself in it, and you will look exhausted. ;o) I'm kidding. Once you practice it a few times, you'll get a knack for it, and it will get easier.