How to deploy vMX with multiple RE’s and multiple FPC’s in EVE-NG (KVM)

As promised (little late but better late than never) I wanted to show you guys an amazing thing, that I recently saw myself at Twitter:
Deploying the vMX with Dual-RE and multiple Linecards (PFE’s) – awesome, right?

For this Setup I followed http://www.eve-ng.net/documentation/howto-s/109-howto-juniper-vmx-16-x-17-x but made slight modifications (added metadata files for each Device) so that both RE’s and all PFE’s would come up correct – it’s obviously not enough to just deploy multiple VM’s 😉

 

Inside your decompressed vMX-Bundle Folder you will find the following files (example for v18.1, click to enlarge):

 

Create folders for VCP-RE0 and VCP-RE1 and also for FPC-0 up to FPC-11 (I personally tried up to 5 due to lack of CPU)  and copy the files (see below example) into the right folders for every fpc and vcp:

 

Don’t forget to fix EVE’s permissions after you have created :
/opt/unetlab/wrappers/unl_wrapper -a fixpermissions

 

When adding nodes to your topology you need to select “RE-0″, RE-1” and your PFE’s – do not add RE0 or PFE0 multiple times – this will not work – for each chassis you can deploy the appropriate VM’s. Connect them through interface “int” – with a bridge this is done quite easy.

Power up and down as many fpc’s as you need – they will now come up with the correct fpc-slot-no and will no longer interfere each other as compared to deploying just fpc0’s which will obviously not work:

By default RE0 will be Master and RE1 will be backup – but like the real MX you can change this via config and request them to failover and so on. Also “commit synchronize” is your friend 😉

 

 

 

Enjoy your “Powerhorse vMX” 😉

25 thoughts on “How to deploy vMX with multiple RE’s and multiple FPC’s in EVE-NG (KVM)

  1. Harinder

    Hey chris,
    I’m trying to setup a lab in which I’m using two vxm’s with dual RE’s and Dual FPC’s on each . I’m not able to ping one router to another. Can you help me on this .
    TIA

    Reply
    1. christianscholz Post author

      Hi,

      sure I can help you with that – can you send me your Setup and configs via chs[at]ip4[dot]de?

      Regards
      Chris

      Reply
  2. Harinder

    Anyway thanks, i found what was the issue.
    how to use service pic’s in vmx17.1 ?

    Reply
  3. sam

    when i login into Juniper site i found multiple image for downloading ( ESXi , KVM , limited , Nested ) and nothing match the eve site ,so which one do you recommended ?

    Reply
    1. christianscholz Post author

      You can use the KVM (qcow2) or the ESXi (and convert them) – both ways work fine

      Reply
  4. sioduy

    hi chris
    i am trying setup vmx with dual RE (RE0 & RE1) and dual PFE (PFE0 & PFE1) i already follow your instruction create each directory RE0 & RE1 run fine, but not with VFP fpc0 & fpc1 not work. after i check pfe0 & pfe1 have same IP address 128.0.0.16

    Reply
    1. christianscholz Post author

      Did you use the files from the Juniper package? Every FPC (pfe) has a different image

      Reply
  5. JustAskin

    Any way to get more than 8 FPC’s running? Past 8 they don’t seem to register correctly, and also tend to crash the RE when you try to reset/query/etc FPC8 and up.

    Reply
  6. Ameen

    For me even a single VCP/VFP is failing. They boot up (After taking ages, even when giving them good resources (16gb ram, 8vcpus) .. but after all that time # show chassis fpc says that slot 0 is unresponsive. Thoughts?

    Reply
    1. christianscholz Post author

      I will try to setup the vMX in a similar environment as you described to check if this error is related to the images or the Hypervisor.
      The Devices run fine on both my Boxes (ESX and Bare-Metal).

      Reply
  7. Luca

    Hi Chris,
    I’ve successfully deployed a vMX with 2 REs and two fpc, but when I tried to use it in a lab, i’ve noted two things:
    1) The fpcX count one less interface than the standard fpc. For ex.: the interface labeled ge-x/0/1 by eve, is seen as ge-x/0/0 by junos.
    2) I can’t ping between devices connected to different fpcs.
    So i’ve tried to bridge all the fpcs ge-x/0/0 togeter. The system started to behave correctly. It seems that the ge-x/0/0 of the “non-slot-zero fpcs” are reserved as “intra-fpc-links”.
    Did you experienced this?

    Reply
    1. christianscholz Post author

      Hi Luca,
      I just connected all “int-Links” together and everything started to work just fine.
      However, I did not test any connectivity at all with other Systems – just a test to see if I could run the vMX with multiple FPC’s.

      Reply
  8. Paul Robinson

    Just thought i’d point out, some versions of vMX seem to have a corrupt metadata-usb-fpc[x].img files (or at least, they dont work properly), but there is an easier alternative. Rather than using the fpc files, boot your vfpc, login as root.

    mkdir -p /var/jnx/card/local
    echo 1 > /var/jnx/card/local/slot

    replace 1 with the fpc you want.

    then “reboot”, i find this a little easier to do when working with eve myself

    Reply
  9. Paul Robinson

    actually, to expand on that previous comment, how I do this is I create a VCP and VFP connected via a bridge, wait for them to boot up then from the vmx cli

    start shell pfe network base-os fpc0 // this drops you to a root linux shell on fpc0
    then
    mkdir -p /var/jnx/card/local/
    echo 1 > /var/jnx/card/local/slot
    reboot

    fpc0 now reboots into fpc1 and while thats happening i create another vfpc, connect it, boot it and it then boots into fpc0… do the same commands again (replacing 1 in the “echo” command with whichever slot you want the fpc to start as) and you can have multiple fpc’s. Reason I find this a little easier is I can use the same fpc image for any number of fpc’s rather than having an image per slot (which gets very crowded in the drop down if your using eve and have multiple vmx versions)

    Reply
  10. Alitaheri

    Hi Chris, I followed your setup and everything comes up as planned. I can see both RE’s and my 5 fpc’s. routing engines gracefully switch over if I disconnect or power down the primary and everything seems to be working just fine. However, I connected this setup to alegacy vSRX, cant ping across. Connected it to a legacy vMX14.1, cant ping across. connected it to a C7200 iol image, cant ping across. Any suggestion?

    Reply
    1. christianscholz Post author

      Hi Alitaheri,

      hard to tell from here – I could successfully connect anything lately when doing this again.
      Could you share your setup? I will look into this.

      BR
      Chris

      Reply
      1. Alitaheri

        Hi Chris, Thank you for your response. To start with, I have 2 RE with 4fpc’s connected through a L2 bridge and everything comes up as far the MX goes. REs see the FPCs and I see the backup RE picking up once the master fails, yet I can not ping any device hooked to any of those FPCs. Im running vMX17.2 on a bare metal server. What other info do you need?

        Reply
  11. Casper

    Hi Chris

    Should one be able to perform hitless ISSU with this setup? Just for testing purposes.

    Reply
    1. christianscholz Post author

      Hi Casper,

      I actually never tried that – but I like the Idea 😉
      Theoretically it should work, but I don’t know if the vMX has a limitation for that.

      BR
      Chris

      Reply
  12. master.cuda

    hi chris
    I received a lot of help from your blog.
    I’ve used the eve-ng tool to do various tests with vMX.
    It consists of routing engine and two FPCs (Service-2G, vFPC) for NAT and Firewall test.
    In VCP, vFPC did not connect after checking the normal connection and pinging the vPC.
    Can you tell me what the problem is?

    Reply
      1. master.cuda

        I tested NAT and Firewall with vMS-MPC in vMX v18.1.

        vFP (vMS-MPC = Service-2G / 4G / 10G module) has confirmed that the problem is not connected to the vPC.
        The problem is that the vPC is connected by vCP (ge-0 / 0/1), so there is no link. This problem is presumed to be a bug in EVE-NG.
        The solution was to set the value of vCP (ge-0 / 0/1) to ge-0 / 0/0 and connect it to the vPC.
          We confirmed that the icmp packet and arp table are normally shared.
        Eve-NG is v2.0.3-95.
        Your blog has helped me a lot. Thank you very much.

        Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Captcha * Time limit is exhausted. Please reload CAPTCHA.