Runaway processes hogging memory and potentially leading to server crash when using startup script

[rub-moodle-its]

System: Ubuntu 18.04.5 LTS
Webserver: Apache Tomcat 8.5.39
24 cores, 8 GB RAM

We're using the following startup script to run our optimised maxima image, since using the command directly has resulted in failure:

#!/bin/bash /var/lib/maximapool/2019090200/maxima-optimised-rub -eval "(cl-user::run)"

We're not sure whether this is the root of the problem we've been experiencing the past days but suspect it might be related.

In irregular intervals, our MaximaPool has produced runaway processes that use up one CPU core an increasingly large amounts of RAM (> 5GB). We guess this is due to syntactically correct but hard to compute requests like those described here:
https://moodle.org/mod/forum/discuss.php?d=278063
We managed to reproduce the behaviour by entering the following example on the MaximaPool status page Test form:
array('AlgEquiv', 'y=(x-a)^5999', 'y=(x-a)^6000', 0, '', '');

Our pool configuration is as follows:

Name	Value
Root directory:	/var/lib/maximapool
Min pool size:	10
Max pool size:	10
Limit on number of processes starting up:	5
Maintenance cycle time:	500 ms
Number of data points for averages:	5
Pool size safety multiplier:	3.0

Values in process.conf for startup & execution timeouts as well as maximum lifetime are on default:

### Maximum lifetimes.
# This is the time that a process is allowed to take when starting up (ms).
startup.timeout = 10000
# This is the time added to the lifetime of a process when it is taken to use
# so that it wont be killed while in use (ms).
execution.timeout = 30000
# This is the lifetime given to a process (ms).
maximum.lifetime = 600000

We currently "solved" the issue by implementing a cronjob that kills all maximapool processes that have been running for more than 2 minutes. This solution is undesirable, though, as it is not very robust - multiple bad requests at the same time might still pose a problem for our server. We're also considering giving the machine more RAM to provide some safety margin.

[aharjula]

Well, you will have trouble if you are just killing processes that have lived over 2 minutes as the point of MaximaPool is to keep processes started up to maximum.lifetime waiting for something to execute. Once something comes in for execution they should be killed after the timeout received from STACK (recommended value is around 2-5s though the tasks that eat memory can eat it very fast) (execution.timelimit is a relic of old times and does not have a role in the current world) unless they finish before that. If you kill the processes yourself based on runtime as opposed to CPU-time then you will probably mess up MaximaPools own process tracking. Off course if you are really running with a pool with max size of 10 then your processes are unlikely to run that long anyway.

Personally in a system with 24 cores and 8GB of ram I would probably up the startup-limit to around 10 and min-pool size to 50 and max to 100, for better load-spike capacity. Though it would not help at all with these expensive processes. Though with that number of cores the memory seems very low, often people tend to set the memory to whole number multiple of cores (hyperthreading included) in gigabytes. Assuming larger minimum pool size the server should not need that many cores anyway and it is often better to have multiple smaller MaximaPool nodes behind a load balancer, which actually helps in this kind of a situation as it is less likely that all parallel nodes receive an overly expensive task and while one node might stuck and waiting for restart by cron or some other means the other can still continue.

MaximaPool itself cannot do anything about those expensive tasks with very large expressions as it does not enforce any memory limit for the tasks and therefore easily ends up in situations where a task could lead to swapping. Adding a memory limit is something that one could probably do with cgroups though. Might be that the next rewrite of MaximaPool will include its own memory limit, could even be that STACK should enforce such a limit itself.

You might also want to look at the version of STACK you have as it is possible that that particular example of expensive input you provided would not cause similar trouble in more recent versions. Though I am aware that the ILIAS version (if you happen to use it) of STACK might not have the latest patches. Note that while STACK now tries to be smarter with input coming in for algebraic equivalence testing it just cannot do everything and a hostile user can still build inputs that can eat up excessive amounts of resources. So when this happens one may want to add some POST logging (to see the payload) to catch the requests that do not complete cleanly and then trace them to originating questions or students, some questions may need to be reformulated to take into account the expense and in extreme cases someone may need to talk to the user/student if they repeatedly sen in such inputs.