[hdiethelm]

The main target of this PR is to simplify the realtime configuration of LinuxCNC

There are three parts:

• scripts/realtime-config-kernel
  • Creates and applies a recommended kernel command line
  • Creates a recommended /etc/sysctl.d/50-linuxcnc-rt.conf
  • Might be more, like blacklisting known bad kernel modules
• scripts/realtime-config-ethernet
  • Configures your network card for realtime
  • Ideally, it is added to /etc/network/interfaces
• scripts/latency-ping + src/hal/components/ping.comp
  • For network testing and tuning
  • Easier and faster to use than LinuxCNC with hal-histogram while the results are representative
• Improve documentation how to configure and test a PC

The CPU detection for SMT / No SMT / Big / Little turned out to be quite complex and there are probably still some bugs.

The documentation is still WIP

[hdiethelm]

@rmu75 Can you run scripts/realtime-config-kernel on your big/little laptop and report the output for all four variants of big/little and SMT on/off? Just to see if it would work, updating grub is not needed.

[rene-dev]

/etc/network/interfaces should not be used when using networkmanager, which is the default. I have seen many problems related to 2 configurations fighting over a network interface. It should be possible to archive the same with nmcli.

[rodw-au]

/etc/network/interfaces should not be used when using networkmanager, which is the default. I have seen many problems related to 2 configurations fighting over a network interface. It should be possible to archive the same with nmcli.

Then you should also update the method discussed in the docs for the hm2_eth driver to set a fixed IP address.
I load non persistent parts of kernel tuning using a startup service to run the script.

Also PREEMPT_RT is misspelt in docs and comments as PREEM_RT

I will review the script as I've done a lot in those space.

[hdiethelm]

Thanks for the hints. This PR is still WIP but i decided to open it allready to collect some inputs.

My preffered solution for network manager until now was just to uninstall it... :-D I need to investigate. Older versions just ignored static interface configs but i havn't had issues since some time.

Having to assign the used CPU manually for network and then hope rtapi_app uses the same will sometimes create issues.

[rodw-au]

Additional optimisations to consider
NIC: gro, tro, lso = off; adaptive_rx =0, rx-frames=0
Set governors to performance
Disable irqbalance
set intel_pstate no_turbo

My standard grub command before adding customized options
"isolcpus=domain,managed_irq,%s nohz_full=%s rcu_nocbs=%s intel_idle.max_cstate=0 processor.max_cstate=1 idle=poll mitigations=off pcie_aspm=off skew_tick=1 intel_pstate=disable"
where %s = isolated core

You may not like to set mitigations due to supposed security risks....

[rodw-au]

Thanks for the hints. This PR is still WIP but i decided to open it allready to collect some inputs.

My preffered solution for network manager until now was just to uninstall it... :-D I need to investigate. Older versions just ignored static interface configs but i havn't had issues since some time.

Having to assign the used CPU manually for network and then hope rtapi_app uses the same will sometimes create issues.

but you need networkmanager so the user can configure wifi if used for general network.
ethtool is no longer installed by default but I use it anyway.
You should be able to read the number of cores and configure the last_core without user involvement, then look for its siblings. running on the lastcore is recommended for preempt_rt

You could also consider sensing network Manufacturer and report if Realtek is used and if R8168-dkms or R8125-dkms drivers are not installed but you'd need to check the list of nics in the Debian repos for those drivers... (I just install them in the ISO by default)

[rmu75]

@rmu75 Can you run scripts/realtime-config-kernel on your big/little laptop and report the output for all four variants of big/little and SMT on/off? Just to see if it would work, updating grub is not needed.

I have to postpone testing until I can get a newer kernel running, the rt preempt debian kernel 6.12.95+deb12 is extremely unstable on the alder lake fujitsu laptop (freezes within a minute after reboot).

[hdiethelm]

@rodw-au
Thanks for the information.

I use cpufreq.off=1 to disable any frequency scaling. This trades of some performance but there is no need to set any governor, the CPU just runs at the base clock. I need to check which solution is better, just disabling all or using governor settings.

You might have to set irqaffinity, otherwise, interrupts still run on the isolated cores. This can be checked by cat /proc/interrupts

NetworkManager is not really needed, you can do all using /etc/network/interfaces. But yes, for most users, this would be really cumbersome.

Just checked: nmcli shows for example enp0s25: unmanaged for all interfaces configured /etc/network/interfaces. I remember having issues for some time for this case. But they went away. So indeed, NetworkManager was fixed, at least on Debian. After updating /etc/network/interfaces, sudo systemctl restart NetworkManager has to be run to update the config. This needs to go to the doc.

Due to the network config script needs root, the solution for NetworkManager only would be dispatchers scripts. But this looks cumbersome:
https://askubuntu.com/questions/1111652/network-manager-script-when-interface-up

A systemd start script will not work, due to the IRQ kernel threads appear only after ifup and disappear after ifdown.

With proper interrupt assignment and priorities, I don't need r8168-dkms anymore. It works also well with the r8169 driver, at least on my PC.

Right now, there are several variants of the command line, automatically set based on sysfs information. Tested so far:
22 core with SMT disabled:
isolcpus=managed_irq,domain,21 irqaffinity=0-20 rcu_nocbs=21 rcu_nocb_poll nohz=on nohz_full=21 intel_idle.max_cstate=1 processor.max_cstate=1 cpufreq.off=1 nosoftlockup nowatchdog nosmt

22 core with SMT on (Tested, you can isolate the SMT sibling, leave SMT on and it works quite well):
isolcpus=managed_irq,domain,21,43 irqaffinity=0-20,22-42 rcu_nocbs=21,43 rcu_nocb_poll nohz=on nohz_full=21,43 intel_idle.max_cstate=1 processor.max_cstate=1 cpufreq.off=1 nosoftlockup nowatchdog

4 core which does not have SMT:
isolcpus=managed_irq,domain,3 irqaffinity=0-2 rcu_nocbs=3 rcu_nocb_poll nohz=on nohz_full=3 intel_idle.max_cstate=1 processor.max_cstate=1 cpufreq.off=1 nosoftlockup nowatchdog

[rene-dev]

apparently nm ignores interfaces in that file:
https://wiki.debian.org/NetworkManager?&pow_referer=#Ignored_network_interfaces

[rene-dev]

@rodw-au

running on the lastcore is recommended for preempt_rt

do you have a source/reason for this?

[hdiethelm]

@rodw-au

running on the lastcore is recommended for preempt_rt

do you have a source/reason for this?

For a long time, rtapi_app run always on the last core, the isolation detection was broken. It was fixed in: #3903 (46c14f9)
Might be this was a workaround for this without ever recognizing that rtapi_app was just running on the wrong core.

You can check this, PSR is the CPU:
ps -L -p "$(pgrep -x rtapi_app)" -o pid,tid,rtprio,psr,pcpu,stat,comm

Isolated CPU 21, master:

    PID     TID RTPRIO PSR %CPU STAT COMMAND
   4616    4616      -  30  0.0 SLsl rtapi_app
   4616    4617      -  11  0.0 SLsl rtapi_app:mesg
   4616    4620     98  21 14.3 SLsl rtapi_app:T#0
   4616    4621     97  21  0.3 SLsl rtapi_app:T#1

On 2.9, it runs on CPU 43 which is just bad:

    PID     TID RTPRIO PSR %CPU STAT COMMAND
   4822    4822      -  34  0.2 SLsl rtapi_app
   4822    4823      -  15  0.0 SLsl rtapi_app:mesg
   4822    4824     98  43 18.7 SLsl rtapi_app:T#0
   4822    4825     97  43  0.5 SLsl rtapi_app:T#1

I wondered why I can only isolate the last CPU and everything else does not work. So I investigated and fixed it...

[rodw-au]

Sorry, I don't have a source. just using last core came up every time I dug into PREEMPT_RT. I was really referring to PREEMPT_RT on AMD64 architecture. There are exceptions like the big.LITTLE architecture on ARM. I could never get good enough performance from my Odroid N2+ with its A73/A53 CPU. In this case, last core puts you on the slower A53 so you need to use a A73 core. I considered getting Xenomai running on it but there was no support for Xenomai on that architecture at that time.

The PR referred to appears to be for Xenomai support so in my mind not really relevant to PREEMPT_RT I've always worked with.

The Realtek DKMS drivers (R8168-dkms, R8125-dkms) were released with Bookworm. The real reason stems from the refusal to include vendor code in the kernel and they managed to get support with the FOSS R8169 driver. My understanding is that the DKMS drivers are building the Realtek vendor code which we don't have to download and build out of tree anymore (because DKMS does it for us). Unless you are an ardent FOSS supporter (I'm not, I just want a machine that works), I would strongly recommend using the DKMS drivers. The Realtek NIC's have enough issues in a RT environment without adding to them. I don't have a source, but its been said the Realtek drivers use 30% more memory than an Intel driver so they need all the help they can get. Wherever possible, follow Beckhoffs lead and avoid Realtek and find a PC with an Intel NIC for RT networked applications (eg Mesa and Ethercat)

I don't think the LinuxCNC community understands that the limiting issue now is network latency not system latency measured by latency-histogram. A standard network stack is adequate for our needs if properly tuned. Linux now supports TSN (time sensitive networking). This requires support in the NIC which is not a major expense but it also needs a matching NIC at the other end (eg the Mesa FGPA or the Ethercat slave). Ethercat does support TSN but I doubt the Ethercat slaves in use for CNC and Mesa FGPA's will ever support it.

[hdiethelm]

The PR referred to appears to be for Xenomai support so in my mind not really relevant to PREEMPT_RT I've always worked with.

This PR is mainly for PREEMPT_RT. But of course, I also add Xenomai support due to I use this most of the time. But I test everything also for PREEMPT_RT.

Right now, PREEMPT_RT on AMD64 but there is no reason to not extend it to other platforms. I have a PI5 lying around waiting to be tested. But this will most probably be a second PR, so this PR is not a never ending project.

R8168-dkms, R8125-dkms

This should go to the docs, i will add it. And possibly detection & warning in the scripts.

I don't think the LinuxCNC community understands that the limiting issue now is network latency not system latency measured by latency-histogram. A standard network stack is adequate for our needs if properly tuned. Linux now supports TSN (time sensitive networking). This requires support in the NIC which is not a major expense but it also needs a matching NIC at the other end (eg the Mesa FGPA or the Ethercat slave). Ethercat does support TSN but I doubt the Ethercat slaves in use for CNC and Mesa FGPA's will ever support it.

Indeed, as long as you don't use the good old parallel port anymore, network is a big contributor to latency. However, if you don't configure the kernel correctly, network latency increases quite a lot due to the network stack is interrupted by other tasks.

[gmoccapy]

This is after running the kernel script and accepting all recommendations.
Commandline is:
linux /boot/vmlinuz-6.12.115+deb13.1-rt-amd64 root=UUID=eb48f77c-de3b-4d80-a6d3-e8ee918280d0 ro isolcpus=managed_irq,domain,5 irqaffinity=0-4 rcu_nocbs=5 rcu_nocb_poll nohz=on nohz_full=5 intel_idle.max_cstate=1 processor.max_cstate=1 cpufreq.off=1 nosoftlockup nowatchdog
This is a Dell Optiplex

gmoccapy@Lathe-optiplex7460aio:~$ inxi -f
CPU:
  Info: 6-core model: Intel Core i5-8500 bits: 64 type: MCP cache: L2: 1.5 MiB
  Speed (MHz): avg: 3001 min/max: N/A cores: 1: 3001 2: 3000 3: 3000 4: 3001
    5: 3004 6: 3000
  Flags: 3dnowprefetch abm acpi adx aes aperfmperf apic arat
    arch_capabilities arch_perfmon art avx avx2 bmi1 bmi2 bts clflush
    clflushopt cmov constant_tsc cpuid cpuid_fault cx16 cx8 de ds_cpl dtes64
    dtherm dts epb erms est f16c flush_l1d fma fpu fsgsbase fxsr ht hwp
    hwp_act_window hwp_epp hwp_notify ibpb ibrs intel_pt invpcid lahf_lm lm
    mca mce md_clear mmx monitor movbe mpx msr mtrr nonstop_tsc nopl nx pae
    pat pbe pcid pclmulqdq pdcm pdpe1gb pebs pge pln pni popcnt pse pse36 pti
    pts rdrand rdseed rdtscp rep_good sdbg sep smap smep smx ss ssbd sse sse2
    sse4_1 sse4_2 ssse3 stibp syscall tm tm2 tsc tsc_adjust
    tsc_deadline_timer vme x2apic xgetbv1 xsave xsavec xsaveopt xsaves
    xtopology xtpr

Can this computer still be otimized?

Nobert

[rodw-au]

That looks disappointing for an i5 but normally I thought they were a 4 core PC. Will you use it with a parallel port? If not add --nobase --sbins 1000 --show as parameters. What PC is it?

[hdiethelm]

Hmm, this looks to me like you are not using a dedicated GPU?

I never had success with intel IGPU's. The only thing that really works are dedicated AMD gpu's. Success so far with an RX460 and an RX 5500 XT. Nvidia is crap a well.

I did not test integrated AMD gpu's yet but do they all share the host memory, I don't expect anything better.

Which load are you running? A sign of a GPU issue is that if you only use stressng and don't touch your mouse, all is fine. As soon as you start glxgears, kicking around, starting firefox or similar, you see spikes.

Anyway, as long as you are using an FPGA card over Ethernet, it should still work well enough.

[gmoccapy]

This is an Dell optiplex7460aio allinone PC.
During the test I was on internet and did some chats with chatgpt and at the same time I was connected with my vdr through ssh.

If you tell not touch a mouse and no gl gears it should be much better, I will try.
It is supposed to be used with a Mesa 7i76e, or may be throw it away😏

[rodw-au]

It actually looks fine for a Mesa card. running this test will clean it up
latency-histogram --nobase --sbins 1000 --show

Most All in Ones I've seen use a Realtek NIC so be sure to
sudo apt install r8169-dkms if this is the case. (need non-free in sources).

[hdiethelm]

This is an Dell optiplex7460aio allinone PC. During the test I was on internet and did some chats with chatgpt and at the same time I was connected with my vdr through ssh.

If you tell not touch a mouse and no gl gears it should be much better, I will try. It is supposed to be used with a Mesa 7i76e, or may be throw it away😏

Much better is not really the case. This experiment is just to identify where the large latency comes from. At the end, you will need your mouse to operate the machine.

However, if you use a Mesa, max. total latency is ~800us where it still works without errors. That means:

• 64.5us thread start latency (measured by latency-historgam)
• Estimating 150-300us network latency (the GPU has also a lot of influence on this, ~70us is what I get with an AMD GPU / 150us with the IGPU)
• ~10 us more latency due to other functions called

So you will end at ~400 us which is not great but good enough.

Can you do me a favor and test the following?

1. Just normal network config, run LinuxCNC
2. Run hal-histogram --minvalue 0 --binsize 2.0e4 --nbins 200 hm2_7i96s.0.read.time
3. Quit LinuxCNC
4. Run sudo scripts/realtime-config-ethernet NIC_NAME 5
5. Run LinuxCNC again
6. Run hal-histogram --minvalue 0 --binsize 2.0e4 --nbins 200 hm2_7i96s.0.read.time

Like 10 min each test should be fine. If you are patient, longer is better.

So I see how well the network config scripts from this PR works on a different machine than I have. NIC_NAME is the name of the interface, you have to adjust it to your PC. 5 the CPU number from your kernel config, should be fine.

You might need to adjust --binsize 2.0e4 to see something useful, if you are using 2.9, it shows clocks, so approx. 2.0e4. On master, it shows time in seconds, so --binsize 0.5e-5 should do the trick.

[rene-dev]

Another idea: i think mitigations=off should have some impact.

[hdiethelm]

Another idea: i think mitigations=off should have some impact.

I tested this some time ago, no visible change as much as I remember. Setting this should mainly decrease the CPU load while making a PC somewhat easy to hack. I will add it to the doc anyway.

[rene-dev]

Another idea: i think mitigations=off should have some impact.

I tested this some time ago, no visible change as much as I remember. Setting this should mainly decrease the CPU load while making a PC somewhat easy to hack. I will add it to the doc anyway.

It should improve performance on older cpus, and is not a problem unless you are running untrusted code. I Think most Linuxcnc maschines are running older cpus, and don't run untrusted code.