And so this series nears its end ! In this article, I will be discussing how the interface to memory management component components will be re-done to better follow the usability conventions of the rest of the kernel, and what this redesign entails from an implementation point of view. Also, may I take the opportunity of this post’s header to urge everyone to do something about the SOPA and PIPA bills that are currently being voted in the USA ? They basically pave the way for legally enforced censorship of the internet by random mega-corporation without due process and on the basis of minor copyright infringement claims.
Physical memory manager (class name : PhyMemManager, instance name : phymem_manager)
This kernel component manages physical RAM. It allocates page-aligned chunks of physical RAM to processes in a fashion that is mutually exclusive as a default, although on-demand sharing can occur. Since the physical memory manager is not supposed to be contacted directly by user-mode processes (see MemAllocator instead for this purpose), its allocation functions return pointers to internal management structures whose content is partially publicly documented instead of specifically crafted “output structures”. This averts the complexity of memory allocation inside of a memory allocator in at least one place.
On architectures of the x86 family, the physical memory manager only allocates physical memory of the high memory region (addresses above 1MB) as a default. This is because low memory is full of important legacy stuff and should not be casually manipulated by unskilled programs. Specific memory allocation routines are available to software which explicitly wants to allocate memory there.
- bool init_malloc() : This function initializes functionality based on dynamic memory allocation. Although the core business of PhyMemManager cannot depend on this it, as it would result in unwanted recursive memory management calls, higher-level functionality such as per-process memory usage counting may need it.
- bool init_process(ProcessManager&) : This function, automatically run by ProcessManager as part of its initialization, sets up everything related to process management in the physical memory manager.
- PID add_process(PID, ProcessProperties) : This function is run by the ProcessManager anytime a new process of specified PID is spawned. It allows the physical memory manager to set up internal management structures about this process, if it needs to. Returns the process’ PID if successful, PID_INVALID otherwise.
- void remove_process(PID) : This function destroys the management structures that are associated to a process and frees all the memory that said process may have allocated.
- PID update_process(PID, PID) : This function is run as part of the live updating of a process. It allows makes the new and the old process switch PIDs, along with possible future physical memory management concepts that outlive a dying process (such as RAM chunks under DMA access).
- PhyMemChunk* alloc_chunk(PID, size_t, bool) : This function allocates a page-aligned chunk of physical memory. It takes the PID of the allocating process as its first parameter, the size of the chunk that is to be allocated as its second parameter (default value : size of a memory page), and allows the user to specify if the chunk should be contiguous or not as its third parameter (default value : false). If successful, it returns a pointer to the chunk’s internal management structure in the kernel address space, which noticeably describes its position, size, owners, and the way it is distributed across physical memory in the case of non-contiguous chunks. This function returns NULL on failure.
- void free_chunk(PID, size_t) : This function frees a previously allocated page-aligned chunk of physical memory from a PID’s grasp. The involved PID is specified, along with the address of the beginning of the chunk. If the memory chunk turns out to have no owner anymore after this operation is applied, it is brought back to the pool of free memory, whereas shared memory chunks just lose their specified owner.
- PhyMemChunk* share_chunk(PID, size_t) : This function allows a PID to become an extra owner of an already allocated chunk whose starting address is specified. It works pretty similarly to alloc_phymem_chunk otherwise.
Virtual memory manager (class name : VirMemManager, instance name : virmem_manager)
The virtual memory manager maps dynamically chunks of physical RAM into a process’ address space. Although designed with paging in mind, its interface should be generic enough to match any kind of virtual memory system. For the rest, most of what has been said about the physical memory manager still applies here : this class is not meant to be contacted directly by user-mode software, and so returns pointers to its management structures as part of its normal behavior. As an extra particularity, the virtual memory manager does not map anything at address 0 so that functionality based on NULL pointers works as expected.
The virtual memory manager depends on the physical memory manager to work.
- bool init_malloc() : Same as above.
- bool init_process(ProcessManager&) : Same as above.
- PID add_process(PID, ProcessProperties) : This function is run by the ProcessManager anytime a new process of specified PID is spawned. It allows the virtual memory manager to set up internal management structures about this process, if it needs to. Returns the process’ PID if successful, PID_INVALID otherwise.
- void remove_process(PID) : Destroys the management structures that are associated to a process and its address space.
- PID update_process(PID, PID) : This function is run as part of the live updating of a process. It allows the new and the old processes to silently switch PIDs.
- VirMemChunk* map_chunk(PID, PhyMemChunk*, VirMemFlags) : This function maps a chunk of physical memory into a process’ address space. The chunk does not have to come from PhyMemManager, for applications like memory-mapped IO it can be fully hand-made by setting the publicly documented members of PhyMemChunk and ignoring the others. The VirMemFlags structure is a set of flags that describes whether the memory chunk should be readable, writable, executable, or global (accessible to all process, useful for kernel data). If the requested flags cannot be applied in hardware, the kernel will choose the closest match that doesn’t break the software at the cost of isolation. This function returns a pointer to the associated management structure if successful, NULL otherwise.
- void free_chunk(PID, size_t) : This function unmaps a chunk of virtual memory from a process’ address space.
- VirMemChunk* adjust_chunk_flags(PID, size_t, VirMemFlags, VirMemFlags) : This function alters the flags of a chunk of virtual memory, for such applications as more secure JIT interpreters in which the runtime-compiled code wouldn’t be able to modify itself (at the cost of a slight interpreter performance hit). Parameters include the location of the chunk of virtual memory, a mask that specifies which virtual memory flags should be altered, and another mask which specifies which values the altered flags should take.
Memory allocator (class name : MemAllocator, instance name : mem_allocator)
The memory allocator is based on the physical and virtual memory managers. It allocates chunks of memory of arbitrary size to processes which need it, including the kernel itself.
The memory allocator depends on the physical memory manager and the virtual memory manager to work.
- There is no public init_malloc() member function here, as initialize functionality based on memory allocation is implicitly performed as part of the memory allocator’s initialization. The memory allocator also sets up the global memory allocation functions used internally by PhyMemManager and VirMemManager.
- bool init_process(ProcessManager&) : Same as above.
- PID add_process(PID, ProcessProperties) : This function is run by the ProcessManager anytime a new process of specified PID is spawned. It allows the memory allocator to set up internal management structures about this process, if it needs to. Returns the process’ PID if successful, PID_INVALID otherwise.
- void remove_process(PID) : Destroys the management structures that are associated to a process and frees all allocated memory.
- PID update_process(PID, PID) : This function is run as part of the live updating of a process. It allows two processes to seamlessly switch PIDs.
- size_t malloc(PID, size_t, VirMemFlags) : This function allocates a chunk of RAM of specified size to a process, mapped in its address space with the access permissions specified in the provided VirMemFlags. It returns the address of the chunk in memory if successful, NULL otherwise.
- size_t malloc_shareable(PID, size_t, VirMemFlags) : This function allocates a chunk of shareable RAM to a process. The difference with regular memory allocation is that the memory allocator makes sure that nothing but this chunk of data will be present in the associated pages of physical and virtual RAM, which is less efficient but allows objects to be safely shared between processes.
- void free(PID, size_t) : This function frees a chunk of RAM from a process. If applied to a shared chunk of memory, it only removes that process’ grasp on the associated RAM, which will only be freed once no process owns it anymore.
- size_t share(PID, size_t, PID, VirMemFlags) : This function makes two processes shares a shareable chunk of RAM. The first parameter is the PID of the “giving” process, the second parameter is the location of the shareable chunk of RAM, the third parameter is the PID of the “receiving” process, and the last parameter specifies how the chunk of RAM should be mapped in the address space of the latter. The returned result is the location of the shared chunk of RAM in the receiver’s address space if the operation is successful, NULL otherwise.
- bool enter_pool(PID, size_t) : In some circumstances, to achieve optimal memory allocation performance, it is desirable to use so-called pooled memory allocation, in which multiple objects are allocated inside of a single chunk of RAM and all freed at once. This function allows a process to enter a memory pool, so that all of its future allocations are made in the memory pool instead of using the regular method. While the memory allocator will check for pool boundaries, it is the process’ responsibility to take care of others issues such as atomicity, pool nesting (which will silently fail if attempted), etc… Returns true if successful, false otherwise.
- size_t leave_pool(PID) : This function allows a process to temporarily or permanently leave pooled memory allocation. The function returns the location at which the memory allocator was within the pool, which can be used to resume pooled memory allocation later.
Associated implementation changes (strikes indicate progress)
- The name of the headers in which memory management components are stored will be adjusted to match their instance name.
- The arch-dependent and arch-agnostic parts of phymem and virmem will be more clearly separated from each other.
- Distinction between page and chunk allocation will be obsoleted.
- Reserved physical memory chunk allocation will be dropped.
- Pointer-based memory chunk manipulation will be dropped from phymem and virmem, when possible.
- Distinctions between freeing and deleting owners will be dropped.
- PhyMemMap and VirMemMap will be renamed PhyMemChunk and VirMemChunk.
Headers will clearly mark the separation between what is standard in the *Chunk types and what isn’t.
- VirMemManager::set_flags will be dropped. It’s an unnecessary compatibility risk.
- VirMemManager and MemAllocator will be deeply overhauled to consider the kernel on a more equal footing with respect to other processes. This involves making VirMemManager silently identity-map kernel memory chunks and build a reliable map of the kernel’s address space on initialization. Also, all of the kernel’s global pages will now be considered when a new process is created. One of the results of this is that the find_thischunk() method of PhyMemManager() will be dropped.
The pooled memory allocation system will also be overhauled to make it more robust, now performing simple checks for pool boundaries and nested pooled allocation (which is unsupported and must thus fail). Finally, it will now be the memory allocator which silently activates kalloc() and other global shortcuts to its functionality as part of its initialization.
And so this ends the redesign of the kernel’s memory management interface. This is not quite, however, the end of the “process abstraction” series that I planned earlier. That is because while I thought about the design of all these kernel components, it appeared to me that two notions were insufficiently well-defined. One is that of live process updating, and the other is that of process properties. Thinking about those, it appeared to me that a more precise definition of these notions implied a partial redesign of all kernel components that have been defined previously, though the changes should hopefully be of small magnitude. More on that in the next, and hopefully last, post of this series.