Monitor your EVE-NG Server via snmp (IPv6 only EVE)

I like the idea of monitoring my EVE-NG Server with my external LibreNMS Server. Setting this up on EVE-NG (Ubuntu 16.04) is relatively easy – even in IPv6 only mode. You just need to install snmpd (apt install snmpd).
However, the description inside the main file is completely wrong for ipv6 in my opinion and you should be aware of that because this is a small detail, that you might miss (I did…).

WARNING: It will discover a freakin LOT of Interfaces – every point-to-point bridge in the background, every little connection will be discovered and monitored by LibreNMS 😀 So be careful – you might kill your box when running large labs…

In the file /etc/snmp/snmpd.conf the team behind the snmpd made a, in my opinion, terrible mistake.

Per default, it listens on udp:161 (so basically EVERY IPv4 Address which is fine, since you should have a firewall in front of your host – different topic…) but just on udp6[::1]:161 which in the IPv4 world would be just 127.0.0.1 – FATAL if you ask me, especially because they state “listen for connections on ALL interfaces” which is clearly wrong. Seems like IPv6 is indeed new even for some senior developers 😉

Here’s my working file (just change public to your desired string and allow snmp through your ipv6 capable firewall):

###############################################################################
#
# EXAMPLE.conf:
#   An example configuration file for configuring the Net-SNMP agent ('snmpd')
#   See the 'snmpd.conf(5)' man page for details
#
#  Some entries are deliberately commented out, and will need to be explicitly activated
#
###############################################################################
#
#  AGENT BEHAVIOUR
#

#  Listen for connections from the local system only
#  Listen for connections on all interfaces (both IPv4 *and* IPv6)
# agentAddress udp:161,udp6:[::1]:161 - commented out because of classic bs, see correct line below

agentAddress udp:161,udp6:161




###############################################################################
#
#  SNMPv3 AUTHENTICATION
#
#  Note that these particular settings don't actually belong here.
#  They should be copied to the file /var/lib/snmp/snmpd.conf
#     and the passwords changed, before being uncommented in that file *only*.
#  Then restart the agent

#  createUser authOnlyUser  MD5 "remember to change this password"
#  createUser authPrivUser  SHA "remember to change this one too"  DES
#  createUser internalUser  MD5 "this is only ever used internally, but still change the password"

#  If you also change the usernames (which might be sensible),
#  then remember to update the other occurances in this example config file to match.



###############################################################################
#
#  ACCESS CONTROL
#

                                                 #  system + hrSystem groups only
view   systemonly  included   .1.3.6.1.2.1.1
view   systemonly  included   .1.3.6.1.2.1.25.1

#view   all         included   .1                80



#rocommunity public default    -V all
rocommunity6 public 




                                                 #  Full access from an example network
                                                 #     Adjust this network address to match your local
                                                 #     settings, change the community string,
                                                 #     and check the 'agentAddress' setting above

                                                 #  Full read-only access for SNMPv3
# rouser   authOnlyUser
                                                 #  Full write access for encrypted requests
                                                 #     Remember to activate the 'createUser' lines above
#rwuser   authPrivUser   priv

#  It's no longer typically necessary to use the full 'com2sec/group/access' configuration
#  r[ow]user and r[ow]community, together with suitable views, should cover most requirements



###############################################################################
#
#  SYSTEM INFORMATION
#

#  Note that setting these values here, results in the corresponding MIB objects being 'read-only'
#  See snmpd.conf(5) for more details
sysLocation    My Location with Spaces
sysContact     My Name with Spaces
                                                 # Application + End-to-End layers
sysServices    72


#
#  Process Monitoring
#
                               # At least one  'mountd' process
proc  mountd
                               # No more than 4 'ntalkd' processes - 0 is OK
proc  ntalkd    4
                               # At least one 'sendmail' process, but no more than 10
proc  sendmail 10 1

#  Walk the UCD-SNMP-MIB::prTable to see the resulting output
#  Note that this table will be empty if there are no "proc" entries in the snmpd.conf file


#
#  Disk Monitoring
#
                               # 10MBs required on root disk, 5% free on /var, 10% free on all other disks
disk       /     10000
disk       /var  5%
includeAllDisks  10%

#  Walk the UCD-SNMP-MIB::dskTable to see the resulting output
#  Note that this table will be empty if there are no "disk" entries in the snmpd.conf file


#
#  System Load
#
                               # Unacceptable 1-, 5-, and 15-minute load averages
load   12 10 5

#  Walk the UCD-SNMP-MIB::laTable to see the resulting output
#  Note that this table *will* be populated, even without a "load" entry in the snmpd.conf file



###############################################################################
#
#  ACTIVE MONITORING
#

                                    #   send SNMPv1  traps
 trapsink     localhost public
                                    #   send SNMPv2c traps
#trap2sink    localhost public
                                    #   send SNMPv2c INFORMs
#informsink   localhost public

#  Note that you typically only want *one* of these three lines
#  Uncommenting two (or all three) will result in multiple copies of each notification.


#
#  Event MIB - automatically generate alerts
#
                                   # Remember to activate the 'createUser' lines above
iquerySecName   internalUser       
rouser          internalUser
                                   # generate traps on UCD error conditions
defaultMonitors          yes
                                   # generate traps on linkUp/Down
linkUpDownNotifications  yes



###############################################################################
#
#  EXTENDING THE AGENT
#

#
#  Arbitrary extension commands
#
 extend    test1   /bin/echo  Hello, world!
 extend-sh test2   echo Hello, world! ; echo Hi there ; exit 35
#extend-sh test3   /bin/sh /tmp/shtest

#  Note that this last entry requires the script '/tmp/shtest' to be created first,
#    containing the same three shell commands, before the line is uncommented

#  Walk the NET-SNMP-EXTEND-MIB tables (nsExtendConfigTable, nsExtendOutput1Table
#     and nsExtendOutput2Table) to see the resulting output

#  Note that the "extend" directive supercedes the previous "exec" and "sh" directives
#  However, walking the UCD-SNMP-MIB::extTable should still returns the same output,
#     as well as the fuller results in the above tables.


#
#  "Pass-through" MIB extension command
#
#pass .1.3.6.1.4.1.8072.2.255  /bin/sh       PREFIX/local/passtest
#pass .1.3.6.1.4.1.8072.2.255  /usr/bin/perl PREFIX/local/passtest.pl

# Note that this requires one of the two 'passtest' scripts to be installed first,
#    before the appropriate line is uncommented.
# These scripts can be found in the 'local' directory of the source distribution,
#     and are not installed automatically.

#  Walk the NET-SNMP-PASS-MIB::netSnmpPassExamples subtree to see the resulting output


#
#  AgentX Sub-agents
#
                                           #  Run as an AgentX master agent
 master          agentx
                                           #  Listen for network connections (from localhost)
                                           #    rather than the default named socket /var/agentx/master
#agentXSocket    tcp:localhost:705

Discussing the IT landscape and impacts of the COVID-19 pandemic

You should check out the Twitter Handles of the Ambassadors soon.
I just had the greatest time discussing the impact on the IT landscape during the COVID-19 pandemic.

This Juniper Networks Ambassador Roundtable featured Michael Marcellin, CMO at Juniper, and from the Ambassadors Stefan Fouant, Chris Parker, Nick Ryce, Paul Clarke, Andrew Alston, Tom Dwyer and myself.
You will soon find it also on this blog 🙂

vSRX 3.0 Template for EVE-NG

Here you find the vSRX 3.0 custom template for your EVE-NG.

Add this file (named vsrx30.yml) to your /opt/unetlab/html/templates/intel/ folder:

<code>
# Copyright (c) 2016, Andrea Dainese
# Copyright (c) 2016, Alain Degreffe
# Copyright (c) 2017, Alain Degreffe
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#     * Redistributions of source code must retain the above copyright
#       notice, this list of conditions and the following disclaimer.
#     * Redistributions in binary form must reproduce the above copyright
#       notice, this list of conditions and the following disclaimer in the
#       documentation and/or other materials provided with the distribution.
#     * Neither the name of the UNetLab Ltd nor  the name of EVE-NG Ltd nor the
#       names of its contributors may be used to endorse or promote products
#       derived from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
---
type: qemu
name: vSRX3-0
cpulimit: 1
icon: JuniperSRX.png
cpu: 2
ram: 4096
eth_name:
- fxp0/mgmt
eth_format: ge-0/0/{0}
ethernet: 6
qemu_nic: virtio-net-pci
console: telnet
qemu_arch: x86_64
qemu_options: -machine type=pc-1.0,accel=kvm -cpu qemu64,+ssse3,+sse4.1,+sse4.2,+x2apic,+aes,pclmulqdq  -serial
  mon:stdio -nographic   -nodefconfig -nodefaults -rtc base=utc
</code>

Don’t forget to include it in your custom_templates file at /opt/unetlab/html/includes/custom_templates.yml:

custom_templates:
- name: JATP
  listname: 'Juniper JATP Appliance (SkyATP on Premise)'
- name: PACKETFENCE
  listname: 'PacketFence OpenSource NAC'
- name: CSRX
  listname: 'Juniper cSRX'
- name: vsrx30
  listname: 'Juniper vSRX3.0'
- name: jspacelogcollector
  listname: 'JunOS Space Log Collector'

Compared to the vSRX-NG Template, which took the vSRX 3.0 6-7min to fully boot, this “new” template (which is basically a linux based template) takes roughly 2-3min to fully boot with 19.4 and 20.1 😉

Enjoy your unleashed vSRX3.0 🙂
Feel free to ask questions / suggestions in the comments here 🙂

EVE-NG Update in IPv6 only environments

The newest eve-ng ( 2.0.6-42(+43+44)-PRO // 2.0.3-108-Community) introduces a lot of cool new features for making your daily work easier than ever, that I will cover in the next few days.

I had some issues while updating my eve-ng (because I run my eve-ng IPv6 only – and yes – v6 ONLY)

If you have v4 only or Dual-Stack, this “problem” is something, that you will not face.
All IPv6 only guys can keep on reading.

During the upgrade, the eve-ng-pro package (I’m sure community does the same) disabled IPv6 since having IPv6 enabled during the update process could break stuff – great for my eve-ng because this way my installation is safe, but bad for my connectivity because if you run IPv6 only like I do – you’ve guessed it – game over for remote access 😀

Here’s how I “rehabilitated” my eve-ng:
Luckily, I have ILO Access to my Server – else, due to COVID-19 and travel “restrictions” this would have been a whole other game…

I opened a Remote Console and logged into eve-ng after giving it some time for the update.
First was a reboot (you should always do this after upgrades).
When it was reachable again, I logged in and looked inside my Syslog file, how ipv6 was disabled.
I found out, that /opt/ovf/ovfstartup.sh disabled ipv6 completely (sysctl -w net.ipv6.conf.all.disable_ipv6=1).

Usually, the script reflects the settings in your /opt/unetlab/html/includes/config.yml (that you edit by setting stuff via web-interface). I’m sure, that this file was changed by the updater from:

#Disable ipv6 on EVE Host
grep -q 'ipv6.*0' /opt/unetlab/html/includes/config.yml 2>/dev/null && sysctl -w net.ipv6.conf.all.disable_ipv6=1
grep -q 'ipv6 ' /opt/unetlab/html/includes/config.yml 2>/dev/null || sysctl -w net.ipv6.conf.all.disable_ipv6=0

to the new “update settings”:

#Disable ipv6 on EVE Host
grep -q 'ipv6.*0' /opt/unetlab/html/includes/config.yml 2>/dev/null && sysctl -w net.ipv6.conf.all.disable_ipv6=1
grep -q 'ipv6 ' /opt/unetlab/html/includes/config.yml 2>/dev/null || sysctl -w net.ipv6.conf.all.disable_ipv6=1

so that regardless of your settings in the web-interface it would make sure, that ipv6 is disabled.

So far so good.

I changed the second line back (grep -q ‘ipv6 ‘ /opt/unetlab/html/includes/config.yml 2>/dev/null || sysctl -w net.ipv6.conf.all.disable_ipv6=0) and rebooted my eve-ng – TADAA – pnet0 has it’s IPv6 address back 🙂

I also modified the script a bit because in IPv6 only networks it would always show you “no ip address on pnet0” on the console instead of your usual “login to http://…..” – after modifying it, it now also shows IPv6 on the console – but the next update will probably break that since I manually edited the file (you shouldn’t do that until you know EXACTLY what you are doing – I repeat – this is NO NO stuff 😛

Here’s what I did:

Change:

IP="$(ifconfig ${INTERFACE} 2> /dev/null | grep 'inet addr' | cut -d: -f2 | cut -d' ' -f1 | grep -E "^[0-9]+.[0-9]+.[0-9]+.[0-9]+$")"

on interface pnet0 to this:

IP="$(ip -o -6 addr show pnet0 | sed -e 's/^.*inet6 \([^ ]\+\).*/\1/' | grep -v ^fe80)"

It still shows /64 behind the URL because I don’t really know how to use awk but hey – it’s a start 😉
The endgame would be to set the URL to https://[2001…..]/ since IPv6 needs braces [ ] in the URL – but that’s just cosmetic.

In the next few days, I will try to upload more Articles and Videos but due to COVID-19, my workload has increased by roughly 900%… Stay Safe guys!

New Video Series is online

You asked for it – now it finally happened: My new Video Series covering EVE-NG with Juniper Gear is online – well at least it will be if youtube finally manages to “prepare” it… I will release the Videos on a weekly / bi-weekly basis (depending on other Projects) and I will cover all questions in the series. Feel free to subscribe, so that you never miss a Video again 🙂 I will also shout via Twitter once the Video is live.


Some of you might wonder what “Netchron” is. Netchron is a Company, that I founded back in 2010. I used it mainly for educational purposes and also for legal reasons. Now, Netchron launches again – this time bigger than ever 😉

Here’s the Link: https://www.youtube.com/watch?v=DYI5_XC1iHE&t=723s

5-Year Juniper Champion

Just realized, that I started to participate in the Champions-Program back in 2015 which makes this year already Special 🙂

Remember to check your status from time to time at https://jpartnertraining.juniper.net/ and refresh your knowledge there. I Manly use the J-Partner-Training to learn new Stuff and for the introduction of new License-Models and other Partner important Topics.

The second benefit is that as Champion you are entitled to Vouchers, that you can claim as a reward every year. On top of that, the Juniper folks also give you access to an O’Reilly Juniper Book (PDF-download) of your choice. I now have 5, because this is also recurring every year along with the Vouchers. This is a great opportunity for partners to certify the employees (at least 2 attempts per participating candidate if they are Ingenious Champions) if they cannot attend a Session where certification is free. Spread the word if you don’t know about this.

Also, your Company gets a nice Banner for the Website:

What are you waiting for? Get (or renew) your Champion Status now

My 2019 Journey and 2020 Goals

Time to reflect the Year 2019 in Terms of Certification and see what 2020 hopefully brings 🙂

2019 was mostly a Re-Certify Year for me because I prepared for my 2 Expert Exams (JNCIE-DC and JNCIE-ENT) and also launched some amazing Projects regarding EVE-NG. I did both first attempts for the E-Exams in 2019 and I am certain, that I will be able to pass them in 2020.


In 2020 in terms of Certification, I will additionally look at LPIC and PaloAlto. Nothing else in 2020, because I was very active for the re-certify process so that I don’t have to re-certify much in 2020 – however 2021 will be hard so I might do some in late 2020, depending on the status then.

My TOP-3 non-certify hot-topics in 2020 are:
IPv6 (gaining even more knowledge)
Python (extremely needed in my opinion if you are a Network-Engineer)
Ansible (out of pure interest)

And as you might have heard, I will build my very own DataCenter in 2020 next to my House. I’m not talking about a room in my house – I’m talking about a separate building with everything a DC needs to have 🙂

What are your Goals for 2020?

EVE-NG in IPv6-only Environments

Shortly before the holidays, the EVE-Team made us all another huge Christmas Present. And I’m not talking about the 30% off for the PRO Edition which you totally should check out on www.eve-ng.net. I’m talking about the ability to run EVE-NG in IPv6-only Environments. YES – it happened 😀

The latest EVE-NG-Pro ( 2.0.6 – 31 ) already introduced the ability to run v6 on the MGMT-Interface of your EVE (manually, but now possible), however, the Pro-Licensing-Server was only reachable via IPv4.
That changed yesterday 🙂

You can now talk to the Lic-Server via IPv4 and IPv6, enabling your EVE-NG-Pro to always fetch a valid license regardless of your IP underlay. This enables a lot of folks the ability to run EVE-NG where only IPv6 is provided (Asia or Germany if you signed up with Unitymedia for example).

Here’s how I “tweaked” my EVE to run in v6-only (Hetzner Server, therefore the Default-Gateway of fe80::1):

# The primary network interface
iface eth0 inet manual
auto pnet0

iface pnet0 inet6 static
  # Main IPv6 Address of the Server
  address 2a01:xxxx:xxxx:xxxx::xxxx
  netmask 64
  gateway fe80::1
  dns-nameservers 2001:4860:4860::8844 2001:4860:4860::8888

As you can see this Server now runs on v6 only via pnet0 (eth0, WAN).
You can access the Web interface via v6, the Server itself can reach out to the License-Server and all Labs run as usual (internally you can of course still use v4).

In the next step, I will tweak the Server so that your internal v4-Addresses-only Hosts will be translated via NAT64 to reach the internet 🙂 That will be posted probably at the end of this week 🙂

I will not try NAT66, because I think that NAT66 is the most stupid Idea ever and breaks the concept of IPv6 in a fundamental way…

I will also try SLAAC with the EVE-NAT-Cloud – with v6 you have so many possibilities and will never run out of addresses again.

I’m a big supporter of IPv6 as you know and running EVE in v6 only is a huge Improvement. Some of you might think “meh – calm down – it’s only another IP Notation”. I can assure you, that it’s far more than that. And if you look at ALL Vendors out there, they all still have a huge way to go until we can live in an IPv6 only world and finally dump v4, NAT and all other evil stuff created out of Address shortage 😛

Happy Holidays