Have you ever noticed that your vSRX, vMX and vQFX run on insane CPU percentage?
Well yes, you might think because on DPDK Hosts (I wrote about that earlier), the v-Devices run in “Poll-Mode”. But that doesn’t mean, you can’t try to “improve” this behavior.
CAUTION: I tried this myself and have not seen any issues so far – however, this does not guarantee fatal possible side-effects, that I’m unaware of or that do not affect my Labs. If you mess with your EVE, you are on your own so try it and if it works fine, if not you should probably change it back or wait for the official release from the EVE-NG Team if this turns out to be working and they feel the need to implement this. I only tried this with “Pure-Juniper” Labs so far.
In my Test-Setup (1 vMX and 2 vQFX) my CPU went from 35% to below 16% (EVE on ESX).
I think Bare (which I will try next) will show the same if not more drops in the CPU usage.
Update 02.09.2019 – Yes. Bare was even more crazy, allowing me to run 40vQFX at the same time at roughly 70% CPU usage!!!! Insane :O
Also, there was no need to reboot the host.
Here’s what I changed:
echo 0 | sudo tee /sys/module/kvm/parameters/halt_poll_ns and
root@eve-ng:~# vim /etc/modprobe.d/kvm.conf (this file will be created)
options kvm halt_poll_ns=0Let me know in the comments or on Twitter if this also worked for you and if you see any side effects please also let me know.
Heres some background in case you wonder, what this does:
https://www.kernel.org/doc/Documentation/virtual/kvm/halt-polling.txt
The interesting Part for me was the Notes at the end:
Care should be taken when setting the halt_poll_ns module parameter as a large value has the potential to drive the cpu usage to 100% on a machine which would be almost entirely idle otherwise.
Sounds like the v-Devices, right? 😉
Hi Christian.
I’ve been looking to get into Juniper and saw your blog. I had a question about setting up Juniper images in EVE. Are you also installing licenses on the vMx images, or will you have limited functionality to do basic testing without the needs of licenses? I see there are evaluation 60-day licenses, so I was just wondering if it was necessary to keep getting these licenses every 60-days to run the images or not.
Thank you!
Hi E.V.
this depends actually what I want to try.
For most of the scenarios I need the 60-day advanced Test License.
After 60 days, you can always “wipe” your vMX, giving you another 60 days (just re-paste your config and license and you’re done).
BR
Chris
Thank you so much Christian! That’s great to know you can just re-use the license by wiping out the image and re-installing a new one.
Hi Christian,
Thanks for the great post. I configured this too in my lab (EVE on ESXi) and saw significant reduction in CPU utilization. Since posting this, have you noticed any side effects from the change? Also I’m curious to know what version of vQFX you’re running in your lab and what resource (vCPU, memory) allocations you’re using. I’m currently running 18.4R1 and EVE defaults to the following which seems generous IMO.
vQFX RE: 2 vCPU, 2G RAM
vQFX PFE: 2 vCPU, 4G RAM
Cheers!
Hi Christian
I just tried this (10 VMX (Junos 14, single image version)… On my server this setup (all sitting at amnesiac) was idle at 14%.. After the changes, it’s idle at 19% 😀
Maybe those older ones don’t use DPDK?
Hi James,
thats odd 😀
I never saw that behavior, but I also don’t use 14 anymore because of all the EVPN Improvements in 18.x
Hi Christine,
Nice to meet you over here. Can I ask you a question about vMX (v14) and (v15) over EVE-NG? Thank you in advanced.
Unable to use telnet to open vMX on EVE-NG. The implementation steps as follows:
1. Download the following two imgs and saved at /opt/unetlab/addons/qemu/;
root@eve-ng:/opt/unetlab/addons/qemu# ls -lrt | grep jinstall
-rw-r–r– 1 root root 992411648 Jun 30 2016 jinstall-vmx-15.1F6.9-domestic.img
-rw-r–r– 1 root root 648740864 Jan 19 01:30 jinstall-vmx-14.1R8.6-domestic.img
root@eve-ng:/opt/unetlab/addons/qemu#
2. create vmx14 and vmx15 directory
root@eve-ng:/opt/unetlab/addons/qemu# mkdir vmx14
root@eve-ng:/opt/unetlab/addons/qemu# mkdir vmx15
root@eve-ng:/opt/unetlab/addons/qemu#
3. convert the img to qcow2:
root@eve-ng:/opt/unetlab/addons/qemu# qemu-img convert -f raw -O qcow2 jinstall-vmx-14.1R8.6-domestic.img vmx14/hda.qcow2
root@eve-ng:/opt/unetlab/addons/qemu# qemu-img convert -f raw -O qcow2 jinstall-vmx-15.1F6.9-domestic.img vmx15/hda.qcow2
root@eve-ng:/opt/unetlab/addons/qemu# chmod -R 777 vmx14/
root@eve-ng:/opt/unetlab/addons/qemu# chmod -R 777 vmx15/
root@eve-ng:/opt/unetlab/addons/qemu# ls -lrt vmx14/
total 633864
-rwxrwxrwx 1 root root 649134080 Jan 19 12:03 hda.qcow2
root@eve-ng:/opt/unetlab/addons/qemu# ls -lrt vmx15/
total 969480
-rwxrwxrwx 1 root root 992804864 Jan 19 12:03 hda.qcow2
root@eve-ng:/opt/unetlab/addons/qemu#
4. Fix permission
root@eve-ng:/opt/unetlab/addons/qemu# /opt/unetlab/wrappers/unl_wrapper -a fixpermissions
5. go to http portal of eve-ng, and setup the vmx router
2 vcpu, 4G RAM for each vMX are configured.
6 start the vmx01 and vmx02 in EVE-NG, however the telnet session looks green but nothing is displayed from either SecureCRT, or Putty.
My question:
1. what setup is wrong? like qemu? junos image? steps?
2. why the telnet over secureCRT or Putty does not show any content? but the telnet session shows green other than red?
3. I have tested eve-ng in bare metal server and in VMware workstation, both do not work. why?
4. vMX VCP and VFP seperated VMs over EVE-NG work very well, but only vmx v14 and v15 cannot be opened by telnet session.
Hi Sam,
1. what setup is wrong? like qemu? junos image? steps?
–> Steps are correct. I looked at the 14 and 15 images on my EVE and they both seem to run fine.
What EVE are you currently using?
2. why the telnet over secureCRT or Putty does not show any content? but the telnet session shows green other than red?
–> It’s showing content here – where did you get the images from?
3. I have tested eve-ng in bare metal server and in VMware workstation, both do not work. why?
–> Hard to tell from remote. Could be a lot of issues
4. vMX VCP and VFP seperated VMs over EVE-NG work very well, but only vmx v14 and v15 cannot be opened by telnet session.
–> That’s because the architecture was different back in the 14.x and 15.x days of the vMX. Juniper later (16.x and onward) split the vMX into a vCP and a vFP. Any Specific reason, why you use the old Versions? Just curious 🙂
BR
Christian
1. eve-ng version:
root@eve-ng:~# dpkg -l eve-ng
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii eve-ng 2.0.3-102 amd64 A new generation software for networking labs.
root@eve-ng:~#
2. it is 60 days trial image from Juniper website. we are Juniper customer
4. because vcp+ vfp are heavy to spend lot vcpu/ram resoure, I’d like a lighter vMX and support most MX feature, so vMX-15 or -14 are preferred in these locations.
Do you think the qemu needs any customized setup? or just default setup?
Hi Chris,
I installed vMX v14 and v15 over EVE-NG, but there is no any content displayed while I run telnet session from Putty or SecureCRT by emulating console login.
I also have the same issue. Could not figure out. Any luck ?
Hi Christine,
I have try to follow your step, but look cpu still high, do I need change other thing ?
Below is my pic’s link.
https://drive.google.com/file/d/1JpYLJSoQPyRyPwUr_93SBYtrPqRxd3Zt/view?usp=sharing
Hi Toro,
thats totally expected on the “top” view.
However, your EVE GUI should reflect the changes and you should see a significant improvement.
Notice, that the latest EVE has this “fix” already in it.
BR
Christian
Hello, is there any possibilty of running nested vmx on top of Eve-NG.. i.e one image having VCP/VFP..
Sure – nested is also available.
Download the qcow2 from the Juniper site and deploy it with a “Linux Template”.
Juniper’s VMX VFPC machine doesn’t boot – On the serial console connection I see the following error messages while boot:
Waiting for root device to be ready…
mount: mounting /dev/sda2 on /mnt failed: No such file or directory
Waiting for root device to be ready…
mount: mounting /dev/sda2 on /mnt failed: No such file or directory
Waiting for root device to be ready…
On some blogs i read solution as “change virtual hard disk(HDD) drive type to IDE for this machine.”
can anyone please help me with the steps to change virtual HDD type to IDE for that VMX machine?
What is your filename for the harddisk for the vMX? virtioa.qcow2 or hda.qcow2?
Christian, thanks for all the great information for using and optimizing our virtual environments. For this particular procedure, if it negatively impacts other virtual platforms (Cisco or Palo Alto in my case), what is the procedure for resuming default/normal behavior? There doesn’t seem to be a default interval specified for the option defined in the created file, so I’m curious whether simply deleting the file will restore normal operation or if halt_poll_ns is a process that would need to be stopped. If this question makes no sense, I apologize. Just want to know how to back out this change.
Deleting the file and rebooting should revert the changes.
WHat issues are you facing with the other images? Are they getting slowed down by the setting?
I have setup eve-ng on a laptop on a separate partition above ubuntu server to have maximum performance out of. The cpu spikes up because of PFE images, I have written a script that limits all qemu proceses at the amount you want so you can run many more vmxs,but use it with care, I found that a value below 70% will do some unexpected results, but anything above is just fine for lab.
Here is the script , it uses cpulimit utillity
#!/bin/bash
echo “performance” | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
rm -r /root/pids.txt
pidof qemu-system-x86_64 >> /root/pids.txt
echo Enter cpu limit in %
read percent
for word in $(cat pids.txt);
do echo $word; cpulimit -b -p $word -l $percent;
done
And here is the cpu utiliz
CPU Utilisation ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │
│—————————+————————————————-+ │
│CPU User% Sys% Wait% Idle|0 |25 |50 |75 100| │
│ 1 55.4 28.5 0.0 16.1|UUUUUUUUUUUUUUUUUUUUUUUUUUUssssssssssssss > | │
│ 2 58.8 25.1 0.0 16.0|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUssssssssssss > | │
│ 3 60.4 23.6 0.0 15.9|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUsssssssssss > | │
│ 4 60.7 24.1 0.0 15.2|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUssssssssssss > | │
│ 5 60.5 23.7 0.0 15.8|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUsssssssssss > | │
│ 6 59.6 25.3 0.0 15.2|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUssssssssssss > | │
│ 7 59.9 24.9 0.0 15.2|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUssssssssssss > | │
│ 8 60.7 23.0 0.0 16.2|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUsssssssssss > | │
│—————————+————————————————-+ │
│Avg 59.5 24.8 0.0 15.7|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUssssssssssss > | │
│—————————+————————————————-+
18
running QEMU nodes
Hi Alin,
sounds interesting. But limiting the amount of CPU would also limit the performance of the boxes?
I will definitely look into your script – thanks a TON for sharing it 🙂