A simple or "baseline" JPEG file is stored as one top-to-bottom scan of the image.  Progressive JPEG divides the file into a series of scans.  The first scan shows the image at the equivalent of a very low quality setting, and therefore it takes very little space.  Following scans gradually improve the quality.  Each scan adds to the data already provided, so that the total storage requirement is roughly the same as for a baseline JPEG image of the same quality as the final scan.  (Basically, progressive JPEG is just a rearrangement of the same data into a more complicated order.)

Ordinary JPEG (left) versus progressive JPEG (right):

Can you see the difference when the image is being loaded/displayed?
You may want to full refresh this page (Ctrl + F5 or F5) or clear your cache and open this page again.
Note: If you have a high-speed Internet connection, the images will probably load too fast for you to see the interlacing or progressive effects.

The advantage of progressive JPEG is that if an image is being viewed on-the-fly as it is transmitted, one can see an approximation to the whole image very quickly, with gradual improvement of quality as one waits longer; this is much nicer than a slow top-to-bottom display of the image.  The disadvantage is that each scan takes about the same amount of computation to display as a whole baseline JPEG file would.  So progressive JPEG only makes sense if one has a decoder that's fast compared to the communication link. (If the data arrives quickly, a progressive-JPEG decoder can adapt by skipping some display passes.  Hence, those of you fortunate enough to have T1 or faster net links may not see any difference between progressive and regular JPEG; but on a modem-speed link, progressive JPEG is great.)

Up until recently, there weren't many applications in which progressive JPEG looked attractive, so it hasn't been widely implemented.  But with the popularity of World Wide Web browsers running over slow modem links, and with the ever-increasing horsepower of personal computers, progressive JPEG has become a win for WWW use. 

Except for the ability to provide progressive display, progressive JPEG and baseline JPEG are basically identical, and they work well on the same kinds of images.  It is possible to convert between baseline and progressive representations of an image without any quality loss.  (But specialized software is needed to do this; conversion by decompressing and recompressing is *not* lossless, due to roundoff errors.)

A progressive JPEG file is not readable at all by a baseline-only JPEG decoder, so existing software will have to be upgraded before progressive
JPEG can be used widely.