+972 52-548-6969
Maximum addressable virtual memory
I wonder what is says about what I am doing right now that I really wish that I could have the OS give me more control over virtual memory allocation. At any rate, the point of this post is to point out something quite important to people writing databases, especially databases that make use of virtual memory a lot.
There isn’t quite as much of it as I thought it would be. Oh, on 32 bits the 4GB limits is really hard to swallow. But on 64 bits, the situation is much better, but still constrained.
On Windows, using x64, you are actually limited to merely 8TB of address space. In Windows 8.1, (and I assume, but couldn’t verify, Windows 2012 R2) you can use up to 128TB of virtual address space. With Linux, at least since 2.6.32, and probably earlier, the limit is 128TB per process.
Implications for Voron, by the way, is that the total size of of all databases in a single process can be up to 8TB (probably somewhat less than that, the other stuff will also need memory). Currently the biggest RavenDB database that I am aware of was approaching the 1.5 – 2.0 TB mark last I checked (several months ago), and Esent, our current technology, is limited to 16TB per database.
So it isn’t great news, but it is probably something that I can live with. And at least I can give proper recommendations. In practice, I don’t think that this would be an issue. But that is good to know.
Comments
Is 32 bit even relevant for you?
You can tell big customers to just use Server 2012. That should solve it for 5-10 years until the next OS version comes out.
Windows has always been lame as a server platform. I'm impressed that they've finally stretched to 128TB in 8.1, it's about time they released an OS that actually supported the full range of the underlying processor. Still, Windows 8 has been so poorly received it hardly matters.
@tobi, Many clients are very resistant to change so they will not switch just because you want it and still ignoring those clients is foolish as this is potential revenue.
I'm not very familiar with Voron (meaning I know 0) but as far as I know you can control virtual memory allocation using VirtualAllocEx this is also known as off heap allocation. The one thing beyond control is the maximum addressable space. In such cases where you are running out you need to move some of those allocated resources to local addressable file that mimics VM, that will hit performance if not done right (you need to know what goes and what stays based on scoring, metrics, requirements etc).
Tobi, 32 bits is relevant pretty much because of _you_. As far as I am aware, you are the one running RavenDB on the most outdated hardware. And yes, you can say that you need Server 2012 for very big DBs, it is just interesting to note those limitations
Bartosz, Voron relies very heavily on memory mapped files. In practice, I expect most 32 bits systems to hit issue when they go beyond 768 MB - 1 GB, just because of the address space limit.
I wonder whether the bottleneck of ESENT is the size of Version Store and transaction size.
See http://managedesent.codeplex.com/discussions/248471 https://blogs.technet.com/b/exchange/archive/2006/04/19/425722.aspx https://groups.google.com/forum/#!topic/ravendb/389Q2D_owbM
The version store for this instance (2) has reached its maximum size of 1023Mb
Ayende, I honestly did not get that comment. I'm not a RavenDB customer. Maybe a confusion with another customer.
@tobi - Wow. There is a "tobi" who posts to the raven forum regularly who runs on very constrained hardware. All this time I thought "blog tobi" was the same as "forum tobi". Lol.
Carsten, No, this has nothing to do about it.
Tobi, See Kijanna's comment. I always assumed that you were the same person from the forum.
One of the benefits of such a banal user name is internet anonymity. If your name is John Smith you are completely anonymous.
Ayende, didn't it seem like a split personality to you? I must have behaved differently than him. My opinions are not exactly mainstream.
@Tobi Not to offend any Tobi ;) But you "sounded" like the "forum Tobi". Ie smart and often a different pov ;)
// Ryan
Just for the record: I'm the other tobi :-)
I'm using RavenDB as an embedded DB for a POS cash register application. The hardware this is running on often only comes with an Atom CPU (D525 and similar). And the OS that is shipped with the hardware mostly is only 32bit.
The worst case I've evere needed to run RavenDB on was a 800Mhz Celeron with 512MB RAM. - it works, but I wouldn't recommend this :-)
@e-tobi, It is good to know that I'm not the only one going down this path. I'm using RavenDB in the same use case (embedded DB for a POS application, with very low-end hardware)... :-)
@Rodirigo: Argh! Another competitor:-) But good to know, I'm not the only one. I think you're only the 3'rd person I'm aware of, that uses RavenDB for a "plain old desktop application" :-) Would be nice to know, what other interesting projects RavenDB is used out there. I'm only aware of some web apps. If you ever want to share experience about using RavenDB in the POS environment, feel free to drop me a mail.
Comment preview