How to deploy vMX with multiple RE’s and multiple FPC’s in EVE-NG (KVM)

As promised (little late but better late than never) I wanted to show you guys an amazing thing, that I recently saw myself at Twitter:
Deploying the vMX with Dual-RE and multiple Linecards (PFE’s) – awesome, right?

For this Setup I followed http://www.eve-ng.net/documentation/howto-s/109-howto-juniper-vmx-16-x-17-x but made slight modifications (added metadata files for each Device) so that both RE’s and all PFE’s would come up correct – it’s obviously not enough to just deploy multiple VM’s 😉

 

Inside your decompressed vMX-Bundle Folder you will find the following files (example for v18.1, click to enlarge):

 

Create folders for VCP-RE0 and VCP-RE1 and also for FPC-0 up to FPC-11 (I personally tried up to 5 due to lack of CPU)  and copy the files (see below example) into the right folders for every fpc and vcp:

 

Don’t forget to fix EVE’s permissions after you have created :
/opt/unetlab/wrappers/unl_wrapper -a fixpermissions

 

When adding nodes to your topology you need to select “RE-0″, RE-1” and your PFE’s – do not add RE0 or PFE0 multiple times – this will not work – for each chassis you can deploy the appropriate VM’s. Connect them through interface “int” – with a bridge this is done quite easy.

Power up and down as many fpc’s as you need – they will now come up with the correct fpc-slot-no and will no longer interfere each other as compared to deploying just fpc0’s which will obviously not work:

By default RE0 will be Master and RE1 will be backup – but like the real MX you can change this via config and request them to failover and so on. Also “commit synchronize” is your friend 😉

 

set system commit synchronize
set chassis redundancy routing-engine 0 master
set chassis redundancy routing-engine 1 backup
set chassis fpc 0 lite-mode
set chassis fpc 1 lite-mode
set chassis fpc 2 lite-mode
set chassis fpc 3 lite-mode
set chassis fpc 4 lite-mode
set chassis fpc 5 lite-mode

 

 

Enjoy your “Powerhorse vMX” 😉

46 thoughts on “How to deploy vMX with multiple RE’s and multiple FPC’s in EVE-NG (KVM)

  1. Harinder

    Hey chris,
    I’m trying to setup a lab in which I’m using two vxm’s with dual RE’s and Dual FPC’s on each . I’m not able to ping one router to another. Can you help me on this .
    TIA

    Reply
    1. christianscholz Post author

      Hi,

      sure I can help you with that – can you send me your Setup and configs via chs[at]ip4[dot]de?

      Regards
      Chris

      Reply
  2. Harinder

    Anyway thanks, i found what was the issue.
    how to use service pic’s in vmx17.1 ?

    Reply
  3. sam

    when i login into Juniper site i found multiple image for downloading ( ESXi , KVM , limited , Nested ) and nothing match the eve site ,so which one do you recommended ?

    Reply
    1. christianscholz Post author

      You can use the KVM (qcow2) or the ESXi (and convert them) – both ways work fine

      Reply
  4. sioduy

    hi chris
    i am trying setup vmx with dual RE (RE0 & RE1) and dual PFE (PFE0 & PFE1) i already follow your instruction create each directory RE0 & RE1 run fine, but not with VFP fpc0 & fpc1 not work. after i check pfe0 & pfe1 have same IP address 128.0.0.16

    Reply
    1. christianscholz Post author

      Did you use the files from the Juniper package? Every FPC (pfe) has a different image

      Reply
  5. JustAskin

    Any way to get more than 8 FPC’s running? Past 8 they don’t seem to register correctly, and also tend to crash the RE when you try to reset/query/etc FPC8 and up.

    Reply
  6. Ameen

    For me even a single VCP/VFP is failing. They boot up (After taking ages, even when giving them good resources (16gb ram, 8vcpus) .. but after all that time # show chassis fpc says that slot 0 is unresponsive. Thoughts?

    Reply
    1. christianscholz Post author

      I will try to setup the vMX in a similar environment as you described to check if this error is related to the images or the Hypervisor.
      The Devices run fine on both my Boxes (ESX and Bare-Metal).

      Reply
  7. Luca

    Hi Chris,
    I’ve successfully deployed a vMX with 2 REs and two fpc, but when I tried to use it in a lab, i’ve noted two things:
    1) The fpcX count one less interface than the standard fpc. For ex.: the interface labeled ge-x/0/1 by eve, is seen as ge-x/0/0 by junos.
    2) I can’t ping between devices connected to different fpcs.
    So i’ve tried to bridge all the fpcs ge-x/0/0 togeter. The system started to behave correctly. It seems that the ge-x/0/0 of the “non-slot-zero fpcs” are reserved as “intra-fpc-links”.
    Did you experienced this?

    Reply
    1. christianscholz Post author

      Hi Luca,
      I just connected all “int-Links” together and everything started to work just fine.
      However, I did not test any connectivity at all with other Systems – just a test to see if I could run the vMX with multiple FPC’s.

      Reply
  8. Paul Robinson

    Just thought i’d point out, some versions of vMX seem to have a corrupt metadata-usb-fpc[x].img files (or at least, they dont work properly), but there is an easier alternative. Rather than using the fpc files, boot your vfpc, login as root.

    mkdir -p /var/jnx/card/local
    echo 1 > /var/jnx/card/local/slot

    replace 1 with the fpc you want.

    then “reboot”, i find this a little easier to do when working with eve myself

    Reply
  9. Paul Robinson

    actually, to expand on that previous comment, how I do this is I create a VCP and VFP connected via a bridge, wait for them to boot up then from the vmx cli

    start shell pfe network base-os fpc0 // this drops you to a root linux shell on fpc0
    then
    mkdir -p /var/jnx/card/local/
    echo 1 > /var/jnx/card/local/slot
    reboot

    fpc0 now reboots into fpc1 and while thats happening i create another vfpc, connect it, boot it and it then boots into fpc0… do the same commands again (replacing 1 in the “echo” command with whichever slot you want the fpc to start as) and you can have multiple fpc’s. Reason I find this a little easier is I can use the same fpc image for any number of fpc’s rather than having an image per slot (which gets very crowded in the drop down if your using eve and have multiple vmx versions)

    Reply
    1. Manaf

      Hi Paul,

      still you need to connect from ge-1/0/1 to use ge-1/0/0

      did you find a solution for this?

      Reply
  10. Alitaheri

    Hi Chris, I followed your setup and everything comes up as planned. I can see both RE’s and my 5 fpc’s. routing engines gracefully switch over if I disconnect or power down the primary and everything seems to be working just fine. However, I connected this setup to alegacy vSRX, cant ping across. Connected it to a legacy vMX14.1, cant ping across. connected it to a C7200 iol image, cant ping across. Any suggestion?

    Reply
    1. christianscholz Post author

      Hi Alitaheri,

      hard to tell from here – I could successfully connect anything lately when doing this again.
      Could you share your setup? I will look into this.

      BR
      Chris

      Reply
      1. Alitaheri

        Hi Chris, Thank you for your response. To start with, I have 2 RE with 4fpc’s connected through a L2 bridge and everything comes up as far the MX goes. REs see the FPCs and I see the backup RE picking up once the master fails, yet I can not ping any device hooked to any of those FPCs. Im running vMX17.2 on a bare metal server. What other info do you need?

        Reply
  11. Casper

    Hi Chris

    Should one be able to perform hitless ISSU with this setup? Just for testing purposes.

    Reply
    1. christianscholz Post author

      Hi Casper,

      I actually never tried that – but I like the Idea 😉
      Theoretically it should work, but I don’t know if the vMX has a limitation for that.

      BR
      Chris

      Reply
  12. master.cuda

    hi chris
    I received a lot of help from your blog.
    I’ve used the eve-ng tool to do various tests with vMX.
    It consists of routing engine and two FPCs (Service-2G, vFPC) for NAT and Firewall test.
    In VCP, vFPC did not connect after checking the normal connection and pinging the vPC.
    Can you tell me what the problem is?

    Reply
      1. master.cuda

        I tested NAT and Firewall with vMS-MPC in vMX v18.1.

        vFP (vMS-MPC = Service-2G / 4G / 10G module) has confirmed that the problem is not connected to the vPC.
        The problem is that the vPC is connected by vCP (ge-0 / 0/1), so there is no link. This problem is presumed to be a bug in EVE-NG.
        The solution was to set the value of vCP (ge-0 / 0/1) to ge-0 / 0/0 and connect it to the vPC.
          We confirmed that the icmp packet and arp table are normally shared.
        Eve-NG is v2.0.3-95.
        Your blog has helped me a lot. Thank you very much.

        Reply
      2. Tuongpham

        Hi Chris,
        What version do you running on your vMX can connect VCP to vMS-MPC(VFP)? kindly reply to me. Thanks bro so much.
        Your blog is very interesting.

        Reply
        1. christianscholz Post author

          Hi Tuongpham,

          I run a lot of Versions from 17.x up to 20.x

          Any specific problem, that you run into?

          Reply
          1. YueSong

            Hi Chris,

            What hardware do you running on your vMX 20.x can connect VCP to vMS-MPC?
            I can’t run it for 18 vCPU 24G memory.

          2. christianscholz Post author

            The vMX (single FPC) can run with 6GB in total, each vFP more adds another 4 GB

  13. Eduardo

    Hello. Do you know if it is possible to deploy multiple FPCs on Vmware ESXi ? Best regards

    Reply
    1. christianscholz Post author

      I have no idea and never tried that – but this sounds fun 🙂
      In theory this should be possible – as long as FPC can talk to VCP

      Reply
  14. Trevor

    i need help on NAT and firewall features. How do we add vFP (vMS-MPC = Service-2G / 4G / 10G module) ? anyone can throw some light on this?

    Reply
  15. shadow

    Hi, I tried using vMX 18.4R2.7 and deploy FPC0, FPC1, and VMS. Even I copy the meta data to the FPC folder, they use the same IP address 128.0.0.16/16 and failed to attach VCP. Any idea? Thanks.

    Reply
  16. Kroniklez

    Hi,

    Anyone here can give more detail how to add vMS-MPC using EVE-NG?

    Thanks and appreciate any help.

    Reply
    1. christianscholz Post author

      If you have the bundle (which is the file you need) there should be a subfolder containing the multiple metadata files (eg. metadata-usb-fpc0.img)

      Reply
  17. Wilson

    Hello Chris

    I’m using vMX (20.3R1.8 ) and ran the process mention above, the REs start without any problem then apply the configuration to put one as master the other one as backup, also the commit synchronize. But I saw that the still appears as independently devices.

    root@vMX-1-RE0> show task replication
    Stateful Replication: Disabled
    RE mode: Not Applicable

    root@vMX-1-RE1# run show task replication
    Stateful Replication: Disabled
    RE mode: Not Applicable

    I’m using the bridge and int (interfaces) to connect the both REs and also the FPCs, there is a command to see if they can see to each other ?

    Regards
    Wilson

    Reply
    1. christianscholz Post author

      “show chassis fpc” should show the other fpc’s if the correct metadata-file is used 🙂

      Reply
  18. lchan

    Thanks Chris for the tutorial! I am using 18.2R1.9 with lite mode and I was able to get dual RE and multiple fpc setup without problem.
    However, I noticed all the fpc port 9 is always down. (ge-0/0/9). I tried set chassis fpc 1 number-of-ports 12 but it didn’t help. Any thoughts?

    Reply
  19. Anthony

    Hi Chris,

    Did you ever manage to get VMX working with performance-mode in eve-ng ?

    It seems to be missing the CPU from the FPC as below, nge091.ref for comparison of a working one.

    root@NGE090.REF> show chassis hardware
    Hardware inventory:
    Item Version Part number Serial number Description
    Chassis VM60FFB8DA7A VMX
    Midplane
    Routing Engine 0 RE-VMX
    CB 0 VMX SCB
    FPC 0 BUILTIN BUILTIN Virtual FPC
    CPU

    root@NGE091.REF> show chassis hardware
    Hardware inventory:
    Item Version Part number Serial number Description
    Chassis VM60F6D4949C VMX
    Midplane
    Routing Engine 0 RE-VMX
    CB 0 VMX SCB
    FPC 0 BUILTIN BUILTIN Virtual FPC
    CPU Rev. 1.0 RIOT-LITE BUILTIN
    MIC 0 Virtual
    PIC 0 BUILTIN BUILTIN Virtual

    Reply
  20. Will Roullier

    Hello Chris,
    Great blog, and learning a lot on this, I do have a slight problem with my setup.
    I have done the exact same thing as yourself exccept I am using a newer version 22.2R3.15
    I have added 2 VFP’s and 2 VCP’s for now to test the functionallity, I have got both REs to work fine, however the 2 VFPs which in this case are fpc0 and fpc1 dont work together. I think I see the issue but I dont quite understand how it works.
    when just turning on fpc0 I can see DHCP requests on the boot logs but they time out and I then presume it defaults to 128.0.0.16 is this normal behaviour? see below:
    ################################################################################
    [ 77.453008] IPv6: ADDRCONF(NETDEV_UP): ext: link is not ready
    Listening on LPF/ext/50:01:00:18:00:00
    Sending on LPF/ext/50:01:00:18:00:00
    Sending on Socket/fallback
    DHCPDISCOVER on ext to 255.255.255.255 port 67 interval 6
    DHCPDISCOVER on ext to 255.255.255.255 port 67 interval 12
    DHCPDISCOVER on ext to 255.255.255.255 port 67 interval 11
    DHCPDISCOVER on ext to 255.255.255.255 port 67 interval 13
    DHCPDISCOVER on ext to 255.255.255.255 port 67 interval 7
    DHCPDISCOVER on ext to 255.255.255.255 port 67 interval 12
    No DHCPOFFERS received.
    No working leases in persistent database – sleeping.
    Ignoring unknown interface int=int.
    done.
    ################################################################################

    If this is normal then can you advise on this. When I then turn on fpc1 it also does the same but also gets the same IP and then both fpcs fail, which is understandable given it has the same IP address, I presumed it worked itself out and the metadata-usb-fpc0 and metadata-usb-fpc1 had different data in it to prevent duplicate assignment? I think this is what the issue I am having but unsure of the right way to fix it without breaking anything.

    Thanks.

    Reply

Leave a Reply to Manaf Cancel reply

Your email address will not be published. Required fields are marked *

Captcha * Time limit is exhausted. Please reload CAPTCHA.