vQFX 10k Testlab on ESX 6.0 / 6.5

Currently the vQFX is neither officially supported for ESX, neither for ESX 6.0 / 6.5.
My Goal is always to have the latest Versions in place – so all the Tutorials for ESX 5.5 are uninteresting for me.

Here are the steps to make the vQFX run on ESX 6.0 / 6.5:

1.) Download the vmdk images from Juniper (RE + PFE)

2.) Upload both files into your datastore

3.) Convert the vmdk images:
vmkfstools -i vqfx10k-re-15.1X53-D60.vmdk vqfx10kRE.vmdk -d thin
vmkfstools -i vqfx10k-pfe-20160609-2.vmdk vqfx10kPFE.vmdk -d thin

4.) Create a new V-Switch for inter-chassis-communication between pfe and re with Promiscious-mode enabled ant MTU of 9000 (Jumbo-Frames)

5.) Create the necessary VM’s:
vQFX-RE:
1 CPU – 2 Cores
5 GB RAM
OS: FreeBSD (64bit)
Adapter: BusLogic – ignore the “not recommended” Warning
Disk: vqfx10kRE.vmdk
Add at least 2 NIC’s:
1st NIC (E1000) – OOB-Management
2nd NIC (E1000) – inter-chassis-communication between PFE and RE
3rd to 10th NIC (E1000) – Data-Links

vQFX-PFE:
1 CPU – 1 Core
2 GB RAM
OS: FreeBSD (64bit)
Adapter: BusLogic – ignore the “not recommended” Warning
Disk: vqfx10kPFE.vmdk
1st NIC (E1000) – OOB-Management
2nd NIC (E1000) – inter-chassis-communication between PFE and RE

6.) Run both VMs

vQFX RE:
login : root
pwd : Juniper

Go to “cli” and configure em0 for OOB-Management.

 

7.) Enjoy – Repeat steps 1-5 for as many Switches as you want 🙂

 

Edit on 16.02.2017:

I want to thank Alexander Marhold for providing a Script that sets the correct mac-adresses to the corresponding interfaces.
http://forums.juniper.net/t5/Ethernet-Switching/vQFX10k-15-1X53-D60-on-ESXi-Installation-and-Running-with-up-to/td-p/303493

I have written a procedure for the vMX and adapted it for the vQFX which does this automatically on each commit

 

The script sets the correct mac address on any configured XE interface  ( taken from the corresponding em(+3) interface,.

  • the mac address is visible under current address in show interface
  • Only if there is a mac adress set in the configuration, that one will be overwritten with the correct one.
  • If the interface belongs to an ae-set, then there will be no mac adress set, as the mac-address is set by the ae
  • if the config contains an interface without a corresponding em-interface , it signals an error on commit

 

Installation on RE

 

> file copy  <location>/set-em-mac-to-xe-ae-vQFX.slax /var/db/scripts/commit/

 

>edit

# set system scripts commit allow-transients

# set system scripts commit file set-em-mac-to-xe-ae-vQFX.slax

#  commit

 

Hope that helps to  install vQFx10k on ESXi, I assume that the mac-seting is also needed on VMware Workstation but I have not tested it.

 

Another hint: there are a bunch of et,xe… interfaces with DHCP in the startup-factory-default, clear them all before starting with your configuration.

and yes independent of your ESXi physical interfaces the interfaces are 10gig XE interfaces.

Download: set-em-mac-to-xe-ae-vQFX.zip

16 thoughts on “vQFX 10k Testlab on ESX 6.0 / 6.5

  1. Any idea why it’s the RE VM that the data plane NICs are added to? In vMX you would add the majority of the NICs to the PFE VM, I would expect RE NICs only to be used for internal communication/obm or in cases where you didn’t want to emulate the PFE.

  2. No idea why. Since the vQFX is a pure “Demo” Case I would expect this to change once the “real”-Version gets out (if it ever gets out).

  3. Hi Christian,

    Your blog is full of information and very easy to understand. I am building my lab in Vmware workstation 14.2 with eve-ng. i have successfully loaded 3xVqfx(re/pfe).
    i am unable to see any any xe-0/0/* interface under >show int terse command, only em interfaces are visible.
    i tried to ran the commit script which you have shared but getting errors. I want to attach files but i am unable to see any attachment option. Thanks

    Regards
    Badar

  4. HI Badar,

    how long dod you wait after boot? Especially the vQFX takes a long time (up to 30mins) until everything is visible. What does “show chassis fpc” indicate? is your fpc online?

  5. Hi christianscholz,

    My broblem is like Badar’s, I just following all steps in home page eve-ng.net, waited 30mins up but show chassis fpc is not visible. Have i need to do more?
    Hope to get response soon,
    Thanks!

    • Hi Vietnamese,
      how did you connect the vRE and the vFPC? are they part of the same “Lan-Segment”?
      Did you change the config so far? Remember, that some parts of the Default-COnfig (for example the em IP’s) are necessary for the RE to communicate with the PFE.

      Regards
      Chris

  6. Hello;
    I am running VQFX on vmware workstation. I am able to see the xe interfaces, however, I am not sure how to put all xe interfaces on the same switching domain so I will be able to ping between them. If I add network adapters for RE, I can see more em interfaces created, however, I am not sure who I can manipulate xe interfaces. Please advise if you have any idea.

    Thanks

    • Hi Al,
      I did not test the vQFX that deep on VMware Workstation, but in ESX there’s also a em Interface per “network-card” and the mapping is 1:1 to the xe-Interfaces.
      Do you mean “Lan Segment” with Switching-Domain”?
      You can also check the MAC-Adresses of the XE / em INterfaces and see if they match with the MAC-Adresses from your WOrkstation Networkcards.

      Regards
      Chris

  7. Hi Chris,

    I got fpc online, now i can see xe- interfaces from xe-0/0/0 to xe-0/0/11. Problem is that i am unable to see anything in ethernet-switching table.

    {master:0}[edit]
    root@MC-Peer-1# run show ethernet-switching table

    {master:0}[edit]

    Please find below the configurations and outputs. even i am able to get an arp entry for irb interface but i think that there is MAC assignment problem on xe-0/0/0.
    how to solve this probelm? how to assign the MAC to xe-0/0/0?

    MC-Peer-1(irb.100-192.168.100.1/30) ————–MC-Peer-2(irb.100-192.168.100.2/30)
    Configurations:

    MC-Peer-1
    set interfaces xe-0/0/0 unit 0 family ethernet-switching interface-mode trunk
    set interfaces xe-0/0/0 unit 0 family ethernet-switching vlan members 100
    set interfaces irb unit 100 family inet address 192.168.100.1/30
    set vlans vlan-100 vlan-id 100
    set vlans vlan-100 l3-interface irb.100

    MC-Peer-2
    set interfaces xe-0/0/0 unit 0 family ethernet-switching interface-mode trunk
    set interfaces xe-0/0/0 unit 0 family ethernet-switching vlan members 100
    set interfaces irb unit 100 family inet address 192.168.100.2/30
    set vlans vlan-100 vlan-id 100
    set vlans vlan-100 l3-interface irb.100

    irb to irb ping successful.

    {master:0}[edit]
    root@MC-Peer-1# run ping 192.168.100.2 rapid
    PING 192.168.100.2 (192.168.100.2): 56 data bytes
    !!!!!
    — 192.168.100.2 ping statistics —
    5 packets transmitted, 5 packets received, 0% packet loss
    round-trip min/avg/max/stddev = 31.887/45.958/87.940/21.081 ms

    {master:0}[edit]
    root@MC-Peer-1# run show vlans 100

    Routing instance VLAN name Tag Interfaces
    default-switch vlan-100 100
    xe-0/0/0.0*

    {master:0}[edit]
    root@MC-Peer-1# run show vlans 100 detail

    Routing instance: default-switch
    VLAN Name: vlan-100 State: Active
    Tag: 100
    Internal index: 4, Generation Index: 4, Origin: Static
    MAC aging time: 300 seconds
    Layer 3 interface: irb.100
    VXLAN Enabled : No
    Interfaces:
    xe-0/0/0.0*,tagged,trunk
    Number of interfaces: Tagged 1 , Untagged 0
    Total MAC count: 0

    {master:0}[edit]
    root@MC-Peer-1# run show ethernet-switching interface xe-0/0/0.0
    Routing Instance Name : default-switch
    Logical Interface flags (DL – disable learning, AD – packet action drop,
    LH – MAC limit hit, DN – interface down,
    MMAS – Mac-move action shutdown,
    SCTL – shutdown by Storm-control )

    Logical Vlan TAG MAC STP Logical Tagging
    interface members limit state interface flags
    xe-0/0/0.0 8192 tagged
    vlan-100 100 1024 Forwarding tagged

    {master:0}[edit]
    root@MC-Peer-1# run show interfaces irb.100
    Logical interface irb.100 (Index 545) (SNMP ifIndex 517)
    Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
    Bandwidth: 1000mbps
    Routing Instance: default-switch Bridging Domain: vlan-100
    Input packets : 182
    Output packets: 182
    Protocol inet, MTU: 1500
    Flags: Sendbcast-pkt-to-re
    Addresses, Flags: Is-Preferred Is-Primary
    Destination: 192.168.100.0/30, Local: 192.168.100.1,
    Broadcast: 192.168.100.3

    {master:0}[edit]
    root@MC-Peer-1# run show arp interface irb.100
    MAC Address Address Name Interface Flags
    02:05:86:71:b2:00 192.168.100.2 192.168.100.2 irb.100 [xe-0/0/0.0] none

    {master:0}[edit]
    root@MC-Peer-1# run show ethernet-switching table

    {master:0}[edit]
    root@MC-Peer-1#

  8. Hi
    even if i assign some address to em3/4/5 then i received below error. it means there is some problem while getting mac address.

    root@MC-Peer-1# set interfaces em4.0 family inet address 6.6.6.6/30

    {master:0}[edit]
    root@MC-Peer-1# commit
    em4error setting host MAC filter table
    em4error setting host MAC filter table
    configuration check succeeds
    commit complete

    what is this error? is this error is creating problem for xe- interfaces?
    please advise.

    Regards
    Badar

    • I wouldn’t set IP’s on the em-Interfaces – all config is done on the xe-Interfaces.
      The only em-interfaces with IP are the “management-em’s” for internal PFE to RE Traffic.

  9. Hi all
    i try to use your recommendations, and successfully load PFE, but RE stop at
    Loading /boot/loader
    _
    BTX loader 1.00 BTX version is 1.02

    i try to use both vqfx10k-re-15.1X53-D60.vmdk and jinstall-vqfx-10-f-18.1R1.9.vmdk
    Any idea?

  10. So i’m using ESXI 6.5 and the biggest thing missed out on here is, when importing the vm’s make sure you set the vm version to version 8. If you do not you will get the following error:

    Loading /boot/loader
    _
    BTX loader 1.00 BTX version is 1.02

    hope that helps!

Leave a Comment

Captcha * Time limit is exhausted. Please reload CAPTCHA.