Large file operations

We once had to develop a feature to basically undo an existing feature, developed in-house, and use a third party code instead. As in any project, the motivation for the same happened to be financial benefits and other reasons beyond technical understanding.

The in-house code was designed to read data from a file (part by part), having encoded information, and to decode the information. It would have been great had the third party application too had been designed similarly. Unfortunately, it was designed to read data only from a buffer and it wouldn't accept any file pointer. So it was evident that someone had to feed this third party code with a pointer to the buffer and moreover it doesn't seem to be a big deal to load the data from the file and store it in a buffer. But here comes our problem, we are in embedded systems and we didn't have the luxury to have a huge buffer in the RAM. But the good news was that the real time operating system was supporting virtual memory and we thought of mapping the file to the memory directly with anonymous mapping mechanism, mmap() does this.

mmap() maps the contents of the file into the memory and returns a pointer to the location. So we are all set to proceed further and passed the pointer to the third party code. Of course, page faults do occur as and when the third party code tries to read more data from the buffer and virtual memory's page swapping does the rest of the job.