+972 52-548-6969
Growable memory
I wish that I could have more control over virtual memory. In particular, what I would really like at this time is the ability to map two virtual address ranges to the same physical memory. And yes, I am well aware (and already using) the ability to do just that using memory mapped files. However, what I would like to do is to have a pure memory based system, without any files being involved. This is meant to be used for the memory only system, which is commonly meant to be be used for unit testing.
The reason that I want to be able to map the physical memory multiple times is that I have a buffer. Let us say that it is 1MB in size. And now I need to grow it. I can’t ask the OS to just grow the buffer, since it might not have the virtual address available past the buffer end by the time I request it. What I would like to do is to request a new virtual allocation, let us say of 2 MB, and then ask the OS to map the same physical memory for the first buffer to the first section of the new buffer, and new memory for the rest.
The end result is that the first part of the buffer is mapped twice, and any changes you make there will be visible in both locations. Now, it is pretty easy to do this with memory mapped files, but I couldn’t find a way to do it sans files.
What I ended up doing is reserve a large portion of the virtual address space (using VirtualAlloc) and then committing it on demand. But I would really have liked to do something better, because now just moved the problem to when I run out of the reserved buffered space.
Comments
But why not use mem mapped files? You don't have to commit them to disk, do you (i mean, as long as you don't run out of memory)?
You can create a file mapping without a file using CreateFileMapping. It is backed by the paging file then, just like any other virtual memory.
I must admit, I also cannot see the benefits of having a (virtual) memory based alternative. A memory map seems to do the job just fine in testing scenarios.
This must be why big sql databases use RAW fs.
VirtualAlloc has no corresponding function in POSIX, how are you going to port this bit to Linux?
So the point of avoiding files is that you want the test to run faster? Why not "cheat" with RAM disk, then. This way the code path will not be different from production (thus testing is more legit), just the performance will be much higher.
When running the OpenLDAP test suite I routinely use a RAMdisk for the test working directory. (tmpfs on Linux.) Saves unnecessary wear and tear on my SSDs.
Some SQL DBs have the notion of temporary tables, which are used to hold intermediate results for a large, complex query. The intent is not to store these on disk, and to have the tables automatically erased after the overall query completes. That's another situation where the DB engine would normally not use a disk file. (It's still moot, since this sort of space is backed by the paging file/swap partition anyway. A lot of what these systems do at higher levels is nonsensical when you look at how it actually works underneath.)
Rafal, Because mem mapped file still have to be allocated to disk. And when you grow them, you need to grow the file on disk, which means that you have to allocate disk space. Which means that you have to interact with the disk in the first place. That is something that we really don't want.
Tobi, While I can create a mem mapped file that is backed by the paging file, that isn't very helpful for growing it. Like any virtual mem, there is no way to specify, give me more memory at location X. With normal files, we can just do a whole new allocation after growing the file size, but that isn't a valid option for page file memory.
Don, Not really, no. Most databases don't really care for the way the file system allocate things. They handle their own internal allocations and space usage internally. In fact, the more layers you have between the disk and the db, the worse we are off. From the point of view of the db, the file system is actually in the way.
Howard, I thought that I would do that using PROT_NONE
Alexei, A RAM disk require the user to install a driver. That isn't usually acceptable for many users.
Howard, We want this to enable specific scenarios for our users, namely quick in memory testing, and they usually don't have RAM disk setup.
In Voron we also have the notion of scratch space. This is used for things like in flight transaction data and during restore operations.
Ayende, that's my point. By using raw fs you can bypass the file system no?
Unfortunately, a lot of OSs lack support for memory map on raw devices.
Not sure why you're so concerned about "memory mapped file still has to be allocated to disk." ALL memory is allocated to disk. That's why it's called virtual memory. If you are worried about memory being allocated to disk, then you shouldn't use VirtualAlloc either. That is also allocated from the page file.
Raymond, You are missing something important. Virtual memory will be paged to disk _as needed_. But with mmap files, you have to first allocate the file on disk.
If you pass INVALID_HANDLE_VALUE to CreateFileMapping, then no file is created. It just creates virtual memory.
As for the issue of growing: Just create a second shared memory block and map it immediately after the first one.
Raymond, That is correct, but that doesn't mean that you can grow that. In effect, you can't increase the size of the mapping, nor can you re-map that section again in a different virtual address. You can hope that you can do another mapping directly after the existing one, but that isn't guaranteed. That is why we need to do reserve & commit.
Raymond, You cannot guarantee that you'll be able to place the next piece of memory immediately after the first one.
Copied from MSDN:
"Non-persisted memory-mapped files
Non-persisted files are memory-mapped files that are not associated with a file on a disk. When the last process has finished working with the file, the data is lost and the file is reclaimed by garbage collection. These files are suitable for creating shared memory for inter-process communications (IPC)."
This sounds like what you want?
JDT, Yes, they aren't growable
As you yourself noted, you don't have the guarantee of placability even if you had growable sections, so the inability to grow in place is no different whether you had growable sections or not And you can map the section to mutliple virtual addresses by calling MapViewOfFile multiple times. I concede that it is more cumbersome finding address space for two consecutive regions since there is no atomic "find me a bunch of contiguous space and map these two separate objects into it".
Raymond, Yes, I know that I can do that. In fact, that is what I do. I grow the file, then I remap the whole size (from start to new end) in another MapViewOfFile. This gives me the behavior that I need, in the sense that both old and new memory are actually pointing in the same physical memory. However, that requires me to physically allocate the file on disk.
Comment preview