File Size and Memory Limitations

The second most common problem, after byte-ordering differences, is the handling of large files. Some systems have limited memory, as is the case with MS-DOS-based, early Macintosh, and other desktop machines. Two problems can arise as a consequence of this limitation. The first is that buffer memory available may not be adequate to handle chunks of data deemed reasonable on a larger machine. Some formats are designed with the limitations of small machines in mind. A prudent thing to do is to avoid forcing the rendering application to buffer more than 32K of data at a time.

Many problems come from hardware limitations, or from interactions between hardware and software. The most notorious example of this is the 64K memory segmentation (known as "chunking" in some environments) on MS-DOS-based Intel machines. Other problems come from software or operating system peculiarities. An example of this is the early Macintosh. This system ostensibly presented a large, flat address space to the programmer. Many programs written with the early Macintosh in mind, however, only handle memory in 32K chunks. Although the trend has been in recent years to restrict programs to 32-bit platforms or higher when there is a choice, data files produced on large machines will continue to be used on smaller machines.

As the speeds of modern machines outstrip some practical limits, such as the speed of main memory and hard disk speeds, the amount of local CPU cache memory becomes more important to system performance. Fast cache memory is relatively expensive compared to main system memory, which tends to encourage system designers to limit the size of cache memory. Nevertheless, it's not uncommon to see a megabyte or more of cache memory in some modern systems. Files can be optimized to make use of this cache, though it's not clear at this time how to make this optimization work across hardware platforms.

The second problem is absolute file size. As suggested above, an uncompressed 24-bit bitmap file 1024 pixels square will be a minimum of 3,145,728 bytes in size. While this much memory may be available on a workstation, it may not be on a smaller machine or on a larger machine in a multitasking situation. In this case, the rendering application will not be able to assemble the data in memory. If any alteration to the image must be done, extraordinary measures may need to be taken by the application prior to rendering. Thus, it is prudent to take advantage of the file "chunking" features available in many formats. Although it may take more programming effort to accommodate smaller machines, the effort also guarantees portability to future platforms.