the job scheduler is slow when upgrading a desktop system

[markjdb]

I have a desktop system that I'm upgrading from FreeBSD:15:amd64 to FreeBSD:16:amd64, about 1000 ports packages are to be upgraded. Once I confirm the upgrade, pkg burns the CPU for a while. Eventually it will finish and start upgrading, but it takes several minutes and the pkg process consumes about 7GB of RAM in the process. (The amount of RAM doesn't change while it's spinning so I presume this is an unrelated problem.)

Some simple profiling with dtrace shows pkg is spending all its time in the scheduler with stacks like this:

              libc.so.7`0x825dc54dc
              pkg`pkg_jobs_schedule_find_cycle+0x7b
              pkg`pkg_jobs_schedule_find_cycle+0x5e
              pkg`pkg_jobs_schedule_find_cycle+0x5e
              pkg`pkg_jobs_schedule_find_cycle+0x5e
              pkg`pkg_jobs_schedule_find_cycle+0x5e
              pkg`pkg_jobs_schedule_find_cycle+0x5e
              pkg`pkg_jobs_schedule+0xbe
              pkg`pkg_jobs_execute+0x91
              pkg`exec_upgrade+0x685
              pkg`main+0xa23
              libc.so.7`__libc_start1+0x12f
              pkg`_start+0x24

That final libc symbol is a strcmp() and there is a missing frame for pkg_jobs_schedule_graph_edge() just below that.

If I enable debug logging, pkg just spits out messages like this for several minutes:

...
DBG(2)[32833]> (scheduler) splitting upgrade libkleo job              
DBG(2)[32833]> (scheduler) splitting upgrade akonadi-contacts job
DBG(2)[32833]> (scheduler) splitting upgrade gcr3 job            
DBG(2)[32833]> (scheduler) splitting upgrade gnupg job        
DBG(2)[32833]> (scheduler) splitting upgrade kgpg job            
DBG(2)[32833]> (scheduler) splitting upgrade kcalc job              

I'm having a lot of trouble getting symbolized stack traces out of pkg at the moment, I will look more in a bit.

[markjdb]

@ifreund this is the problem @bapt and I were talking about on Thursday.

[ifreund]

I think that this will be addressed by #2547, which should significantly reduce the amount of cycles in the job graph and therefore the amount of work done by the scheduler.

[ifreund]

#2547 reportedly took the time spent in the scheduler from several minutes down to around 5 seconds, which makes this issue more of an annoyance than a blocker.

I do have ideas for how to eliminate this pause, so I'll leave this open for now.