vQFX 10k Testlab on ESX 6.0 / 6.5

Currently the vQFX is neither officially supported for ESX, neither for ESX 6.0 / 6.5.
My Goal is always to have the latest Versions in place – so all the Tutorials for ESX 5.5 are uninteresting for me.

Here are the steps to make the vQFX run on ESX 6.0 / 6.5:

1.) Download the vmdk images from Juniper (RE + PFE)

2.) Upload both files into your datastore

3.) Convert the vmdk images:
vmkfstools -i vqfx10k-re-15.1X53-D60.vmdk vqfx10kRE.vmdk -d thin
vmkfstools -i vqfx10k-pfe-20160609-2.vmdk vqfx10kPFE.vmdk -d thin

4.) Create a new V-Switch for inter-chassis-communication between pfe and re with Promiscious-mode enabled ant MTU of 9000 (Jumbo-Frames)

5.) Create the necessary VM’s:
vQFX-RE:
1 CPU – 2 Cores
5 GB RAM
OS: FreeBSD (64bit)
Adapter: BusLogic – ignore the “not recommended” Warning
Disk: vqfx10kRE.vmdk
Add at least 2 NIC’s:
1st NIC (E1000) – OOB-Management
2nd NIC (E1000) – inter-chassis-communication between PFE and RE
3rd to 10th NIC (E1000) – Data-Links

vQFX-PFE:
1 CPU – 1 Core
2 GB RAM
OS: FreeBSD (64bit)
Adapter: BusLogic – ignore the “not recommended” Warning
Disk: vqfx10kPFE.vmdk
1st NIC (E1000) – OOB-Management
2nd NIC (E1000) – inter-chassis-communication between PFE and RE

6.) Run both VMs

vQFX RE:
login : root
pwd : Juniper

Go to “cli” and configure em0 for OOB-Management.

 

7.) Enjoy – Repeat steps 1-5 for as many Switches as you want 🙂

 

Edit on 16.02.2017:

I want to thank Alexander Marhold for providing a Script that sets the correct mac-adresses to the corresponding interfaces.
http://forums.juniper.net/t5/Ethernet-Switching/vQFX10k-15-1X53-D60-on-ESXi-Installation-and-Running-with-up-to/td-p/303493

I have written a procedure for the vMX and adapted it for the vQFX which does this automatically on each commit

 

The script sets the correct mac address on any configured XE interface  ( taken from the corresponding em(+3) interface,.

  • the mac address is visible under current address in show interface
  • Only if there is a mac adress set in the configuration, that one will be overwritten with the correct one.
  • If the interface belongs to an ae-set, then there will be no mac adress set, as the mac-address is set by the ae
  • if the config contains an interface without a corresponding em-interface , it signals an error on commit

 

Installation on RE

 

> file copy  <location>/set-em-mac-to-xe-ae-vQFX.slax /var/db/scripts/commit/

 

>edit

# set system scripts commit allow-transients

# set system scripts commit file set-em-mac-to-xe-ae-vQFX.slax

#  commit

 

Hope that helps to  install vQFx10k on ESXi, I assume that the mac-seting is also needed on VMware Workstation but I have not tested it.

 

Another hint: there are a bunch of et,xe… interfaces with DHCP in the startup-factory-default, clear them all before starting with your configuration.

and yes independent of your ESXi physical interfaces the interfaces are 10gig XE interfaces.

Download: set-em-mac-to-xe-ae-vQFX.zip

23 thoughts on “vQFX 10k Testlab on ESX 6.0 / 6.5

  1. Jerrad

    Any idea why it’s the RE VM that the data plane NICs are added to? In vMX you would add the majority of the NICs to the PFE VM, I would expect RE NICs only to be used for internal communication/obm or in cases where you didn’t want to emulate the PFE.

    Reply
  2. christianscholz Post author

    No idea why. Since the vQFX is a pure “Demo” Case I would expect this to change once the “real”-Version gets out (if it ever gets out).

    Reply
  3. Badar

    Hi Christian,

    Your blog is full of information and very easy to understand. I am building my lab in Vmware workstation 14.2 with eve-ng. i have successfully loaded 3xVqfx(re/pfe).
    i am unable to see any any xe-0/0/* interface under >show int terse command, only em interfaces are visible.
    i tried to ran the commit script which you have shared but getting errors. I want to attach files but i am unable to see any attachment option. Thanks

    Regards
    Badar

    Reply
  4. christianscholz Post author

    HI Badar,

    how long dod you wait after boot? Especially the vQFX takes a long time (up to 30mins) until everything is visible. What does “show chassis fpc” indicate? is your fpc online?

    Reply
  5. Vietnamese

    Hi christianscholz,

    My broblem is like Badar’s, I just following all steps in home page eve-ng.net, waited 30mins up but show chassis fpc is not visible. Have i need to do more?
    Hope to get response soon,
    Thanks!

    Reply
    1. christianscholz Post author

      Hi Vietnamese,
      how did you connect the vRE and the vFPC? are they part of the same “Lan-Segment”?
      Did you change the config so far? Remember, that some parts of the Default-COnfig (for example the em IP’s) are necessary for the RE to communicate with the PFE.

      Regards
      Chris

      Reply
  6. Al

    Hello;
    I am running VQFX on vmware workstation. I am able to see the xe interfaces, however, I am not sure how to put all xe interfaces on the same switching domain so I will be able to ping between them. If I add network adapters for RE, I can see more em interfaces created, however, I am not sure who I can manipulate xe interfaces. Please advise if you have any idea.

    Thanks

    Reply
    1. christianscholz Post author

      Hi Al,
      I did not test the vQFX that deep on VMware Workstation, but in ESX there’s also a em Interface per “network-card” and the mapping is 1:1 to the xe-Interfaces.
      Do you mean “Lan Segment” with Switching-Domain”?
      You can also check the MAC-Adresses of the XE / em INterfaces and see if they match with the MAC-Adresses from your WOrkstation Networkcards.

      Regards
      Chris

      Reply
  7. Badar

    Hi Chris,

    I got fpc online, now i can see xe- interfaces from xe-0/0/0 to xe-0/0/11. Problem is that i am unable to see anything in ethernet-switching table.

    {master:0}[edit]
    root@MC-Peer-1# run show ethernet-switching table

    {master:0}[edit]

    Please find below the configurations and outputs. even i am able to get an arp entry for irb interface but i think that there is MAC assignment problem on xe-0/0/0.
    how to solve this probelm? how to assign the MAC to xe-0/0/0?

    MC-Peer-1(irb.100-192.168.100.1/30) ————–MC-Peer-2(irb.100-192.168.100.2/30)
    Configurations:

    MC-Peer-1
    set interfaces xe-0/0/0 unit 0 family ethernet-switching interface-mode trunk
    set interfaces xe-0/0/0 unit 0 family ethernet-switching vlan members 100
    set interfaces irb unit 100 family inet address 192.168.100.1/30
    set vlans vlan-100 vlan-id 100
    set vlans vlan-100 l3-interface irb.100

    MC-Peer-2
    set interfaces xe-0/0/0 unit 0 family ethernet-switching interface-mode trunk
    set interfaces xe-0/0/0 unit 0 family ethernet-switching vlan members 100
    set interfaces irb unit 100 family inet address 192.168.100.2/30
    set vlans vlan-100 vlan-id 100
    set vlans vlan-100 l3-interface irb.100

    irb to irb ping successful.

    {master:0}[edit]
    root@MC-Peer-1# run ping 192.168.100.2 rapid
    PING 192.168.100.2 (192.168.100.2): 56 data bytes
    !!!!!
    — 192.168.100.2 ping statistics —
    5 packets transmitted, 5 packets received, 0% packet loss
    round-trip min/avg/max/stddev = 31.887/45.958/87.940/21.081 ms

    {master:0}[edit]
    root@MC-Peer-1# run show vlans 100

    Routing instance VLAN name Tag Interfaces
    default-switch vlan-100 100
    xe-0/0/0.0*

    {master:0}[edit]
    root@MC-Peer-1# run show vlans 100 detail

    Routing instance: default-switch
    VLAN Name: vlan-100 State: Active
    Tag: 100
    Internal index: 4, Generation Index: 4, Origin: Static
    MAC aging time: 300 seconds
    Layer 3 interface: irb.100
    VXLAN Enabled : No
    Interfaces:
    xe-0/0/0.0*,tagged,trunk
    Number of interfaces: Tagged 1 , Untagged 0
    Total MAC count: 0

    {master:0}[edit]
    root@MC-Peer-1# run show ethernet-switching interface xe-0/0/0.0
    Routing Instance Name : default-switch
    Logical Interface flags (DL – disable learning, AD – packet action drop,
    LH – MAC limit hit, DN – interface down,
    MMAS – Mac-move action shutdown,
    SCTL – shutdown by Storm-control )

    Logical Vlan TAG MAC STP Logical Tagging
    interface members limit state interface flags
    xe-0/0/0.0 8192 tagged
    vlan-100 100 1024 Forwarding tagged

    {master:0}[edit]
    root@MC-Peer-1# run show interfaces irb.100
    Logical interface irb.100 (Index 545) (SNMP ifIndex 517)
    Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
    Bandwidth: 1000mbps
    Routing Instance: default-switch Bridging Domain: vlan-100
    Input packets : 182
    Output packets: 182
    Protocol inet, MTU: 1500
    Flags: Sendbcast-pkt-to-re
    Addresses, Flags: Is-Preferred Is-Primary
    Destination: 192.168.100.0/30, Local: 192.168.100.1,
    Broadcast: 192.168.100.3

    {master:0}[edit]
    root@MC-Peer-1# run show arp interface irb.100
    MAC Address Address Name Interface Flags
    02:05:86:71:b2:00 192.168.100.2 192.168.100.2 irb.100 [xe-0/0/0.0] none

    {master:0}[edit]
    root@MC-Peer-1# run show ethernet-switching table

    {master:0}[edit]
    root@MC-Peer-1#

    Reply
    1. christianscholz Post author

      I don’t know what exactly went wrong – I can see MAC’s on my vQFX…

      Reply
  8. Badar

    Hi
    even if i assign some address to em3/4/5 then i received below error. it means there is some problem while getting mac address.

    root@MC-Peer-1# set interfaces em4.0 family inet address 6.6.6.6/30

    {master:0}[edit]
    root@MC-Peer-1# commit
    em4error setting host MAC filter table
    em4error setting host MAC filter table
    configuration check succeeds
    commit complete

    what is this error? is this error is creating problem for xe- interfaces?
    please advise.

    Regards
    Badar

    Reply
    1. christianscholz Post author

      I wouldn’t set IP’s on the em-Interfaces – all config is done on the xe-Interfaces.
      The only em-interfaces with IP are the “management-em’s” for internal PFE to RE Traffic.

      Reply
  9. Siddiqui

    I am trying to follow the same, but my RE of VQFX stucks in db> mode . Any suggestion ??

    Reply
    1. christianscholz Post author

      db-mode sounds like a corrupted image – did you try to download it again?

      Reply
  10. Pavel M

    Hi all
    i try to use your recommendations, and successfully load PFE, but RE stop at
    Loading /boot/loader
    _
    BTX loader 1.00 BTX version is 1.02

    i try to use both vqfx10k-re-15.1X53-D60.vmdk and jinstall-vqfx-10-f-18.1R1.9.vmdk
    Any idea?

    Reply
  11. Omar

    So i’m using ESXI 6.5 and the biggest thing missed out on here is, when importing the vm’s make sure you set the vm version to version 8. If you do not you will get the following error:

    Loading /boot/loader
    _
    BTX loader 1.00 BTX version is 1.02

    hope that helps!

    Reply
  12. Andre

    Hi Christian,

    Thank you for this information. I used it to configure a lab of vMXs and vQFXs. I downloaded the script you created for the qfx on vmware esxi, and boom started working. I thought my troubles were over; however I am having an issue with vMX talking the the vQFX. I can get a vMX -to-vMX to talk, and a vQFX-to-vQFX to talk but that’s it.
    Would you be able to give insight to this issue, or point me in the right direction.

    Reply
    1. christianscholz Post author

      Hi Andre,

      how should they talk to each other? L2 or L3?
      What did you configure and how are they “physically” connected?
      Promiscuous mode in the vSwitches on ESX active?

      BR
      Chris

      Reply
  13. Andre A Rogers

    Thanks for the reply. I did a virtual switch in promiscuous mode, and mtu at 9000. I tried layer 3 first to test a simple ping. But is it possible to have both l2, and l3 capabilities?

    Reply
  14. Andre A Rogers

    Sorry read my post, and I don’t think I answered your question well. I can elaborate better with bullet points…. Lol
    • VMware ESXI 6.5
    o vMX version 16.1R7
     VCP and VFP have interconnects, and communicating fine.
    o Vqfx version 15.1×53
     Re and pfe has the same interconnect settings as the vmx
    Creating a topology that has a lot of vMX and vQFX interconnects.
    Tested for simple pings at first…
    • vMX — to — vMX works fine.
    • vQFX – to – vQFX works
    • vMX –to vQFX doesn’t work. On a vSwitch between host
    Further testing was a vSwitch(with physical port), connected to a vMX on Host1. And vSwitch(with physical port), connected to a vQFX on Host2, works with a simple ping. But not a vlan trunk with a l3-interface.
    Hope I am making sense. Sorry have been working on this for a few days. Any insight would be appreciated.

    Reply
    1. christianscholz Post author

      Hi Andre,

      if L3 is working, then L2 must also be working – my guess is, that you need to configure a bridge-domain and associate the irb interface with it.
      What config are you using? Can you send me your files via Mail? I will look into it and test it.

      BR
      Chris

      Reply

Leave a Reply to Badar Cancel reply

Your email address will not be published. Required fields are marked *

Captcha * Time limit is exhausted. Please reload CAPTCHA.