On “asymmetric” memory sharing

Memory sharing is a tricky matter. When done right, it makes it easy for several processes to share access to a common data set rather than wasting CPU time and RAM making thousands of copies of it. But since it pokes a hole in process isolation, it can also allow processes to harm each other in unwanted ways if implemented improperly. In this post, I will discuss decisions which have been made in the past in this area, and some things which I’m going to change in the near future.

In some shared memory system, all processes which share a memory chunk are not treated equally : one process, generally the one which allocated the chunk, keeps all memory management rights on it, while other processes are only given the option to peek into a carefully chosen part of his address space. This policy is fairly easy to implement, and suitable for some scenarios like shared libraries implementation, but it also puts a lot of trust on the allocating process, basically allowing it to segfault all processes with which it shares memory simply by freeing the memory chunk. For this reason, I consider such a sharing mechanism to be flawed from a process isolation point of view.

Instead, I have so far chosen to go in the opposite direction, treating all processes which share a memory chunk in a wholly symmetrical way. In the specific case of memory allocation and liberation, this policy entails that when one process “frees up” a shared chunk, it only removes it from its own address space, while physical memory liberation will wait until references to the shared chunk have been eliminated from all processes. So far, so good.

But sometimes, one wants processes to have unequal access rights on a memory chunk. As an example, in the case of shared libraries, the library code should be mapped as read-only memory regions in user processes’ address space, and user processes should be subsequently unable to adjust these access flags to acquire write access on said code. Otherwise, library corruption can occur across multiple processes, which again breaches process isolation.

The consequence of this is that we want some to put some kind of immutable write access flag on the memory regions associated to shared libraries (and other read-only system objects). And the way I plan to do this is to alter the whole paging mechanism so that it basically accepts two sets of page flags, one which actually controls paging behaviour, and one which acts as a mask marking which page flags the owner process is allowed to alter.

The reason why I propose to do it in such a way is that I can also envision situations outside of the realm of sharing where a process may want to have immutable access to some regions of memory, regardless of sharing considerations. As an example, most processes do not need to modify their own code, and the ability to do so is an attack vector frequently exploited by malware, so they should be forbidden as a default to create RWX regions or to turn RW- regions into R-X regions. But interpreters, which do need to perform such operations sometimes, would probably want to separate “unsafe” memory regions, where code generation occurs, from “safe” memory regions, where the core interpreter code is located, so as to reduce the risk of exploits. And a general mechanism for page flags immutability could be used to do just that, in addition to the “limited sharing” scenario proposed above.

And I guess that’s all, folks !

4 thoughts on “On “asymmetric” memory sharing

  1. Hadrien January 24, 2013 / 7:37 pm

    I prefer to live in a word where NUMA doesn’t exist for now, it’s just too much of a hassle, although I guess one can somewhat deal with it using strong thread-processor affinity :)

    Desktop and laptop processors have been fast enough for so long that I can’t imagine the kind of massively multicore devices that require NUMA to leave the realm of HPC and enter the average personal computer. However, I have to admit that I’m pretty sure that the same has been said about multicore chips before…

  2. Alfman January 27, 2013 / 5:46 am

    “I prefer to live in a word where NUMA doesn’t exist for now, it’s just too much of a hassle, although I guess one can somewhat deal with it using strong thread-processor affinity :)”

    I’ll assume you meant process-processor affinity because each thread within a process would be extremely likely to need to access the same memory, which would be suboptimal from a “foreign” CPU. Personally I’d treat NUMA as a cluster of independent systems with an exceptionally fast interconnect between them, but like you say I’m not sure it has any relevancy for you.

    “Desktop and laptop processors have been fast enough for so long that I can’t imagine the kind of massively multicore devices that require NUMA to leave the realm of HPC and enter the average personal computer.”

    Of course we agree that NUMA doesn’t matter for what you are doing with TOSP, but I don’t think you need to look at “massively multicore” devices to find evidence of the bottlenecks which NUMA addresses. You might remember the program I sent you last year showing a memory intensive program performing best using one thread even though it was theoretically 100% parallel (I hope I’m remembering that correctly, it came up the the context of async vs threaded IO).

    All SMP benchmarks show diminishing returns with more threads, which is often attributed to the proportion of a program which is serial versus parallel in nature as described by Amdahl’s law. It overlooks that even programs which are 100% parallel will incur diminishing returns on SMP systems due to memory bottlenecks, I haven’t found much literature on the subject.

    http://en.wikipedia.org/wiki/Amdahl%27s_law

    For the purposes of TOSP, I think your design philosophy should be “Assuming we had infinite computing power, how could we use it to help the user?” Today’s concepts of technology should not get in the way of the vision (whatever it may be).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s