.NET CORE and .NET 5
Because the question is about .Net Core I must include why going AppDomain route won't work.
App Domains
Why was it discontinued? AppDomains require runtime support and are generally quite expensive. While still implemented by CoreCLR, it’s not available in .NET Native and we don’t plan on adding this capability there.
What should I use instead? AppDomains were used for different purposes. For code isolation, we recommend processes and/or containers. For dynamic loading of assemblies, we recommend the new AssemblyLoadContext class.
Source: Porting to .NET Core | .NET Blog
This leaves us only one way to do this if you want to have "automatic" information about memory usage.
How to measure memory usage for a separate process
To measure other process memory we can use the Process handle and get its WorkingSet, PrivateMemory, and VirtualMemory. More about memory types
The code to handle another process is quite simple.
private Process InterpreterProcess;
// Run every how often you want to check for memory
private void Update()
{
var workingSet = InterpreterProcess.WorkingSet64;
if(workingSet > Settings.MemoryAllowed)
{
InterpreterProcess.Kill(true);
}
}
private void Start()
{
InterpreterProcess = new Process(...);
// capture standard output and pass it along to user
Task.Run(() =>
{
Update();
Thread.Sleep(50);
// This will be also convenient place to kill process if we exceeded allowed time
});
}
This, however, leaves us with a very important question, since we might allow users to access system critical resources - even if we do not run Administrator privileges on the Process.
Alternative approach
Since you mentioned that you have a custom interpreter it might be easier for you to add memory management, memory counting, and security to the interpreter.
Since we can assume that memory allocation is only made with new your interpreter needs to just test the size of each new allocation and test accordingly.
To test managed object size you need to "analyze" created instance and use sizeof() on each simple type, a proper way is in the making There's no way to get the managed object size in memory #24200
