As promised (little late but better late than never) I wanted to show you guys an amazing thing, that I recently saw myself at Twitter:
Deploying the vMX with Dual-RE and multiple Linecards (PFE’s) – awesome, right?
For this Setup I followed http://www.eve-ng.net/documentation/howto-s/109-howto-juniper-vmx-16-x-17-x but made slight modifications (added metadata files for each Device) so that both RE’s and all PFE’s would come up correct – it’s obviously not enough to just deploy multiple VM’s 😉
Inside your decompressed vMX-Bundle Folder you will find the following files (example for v18.1, click to enlarge):
Create folders for VCP-RE0 and VCP-RE1 and also for FPC-0 up to FPC-11 (I personally tried up to 5 due to lack of CPU) and copy the files (see below example) into the right folders for every fpc and vcp:
Don’t forget to fix EVE’s permissions after you have created :
/opt/unetlab/wrappers/unl_wrapper -a fixpermissions
When adding nodes to your topology you need to select “RE-0″, RE-1” and your PFE’s – do not add RE0 or PFE0 multiple times – this will not work – for each chassis you can deploy the appropriate VM’s. Connect them through interface “int” – with a bridge this is done quite easy.
Power up and down as many fpc’s as you need – they will now come up with the correct fpc-slot-no and will no longer interfere each other as compared to deploying just fpc0’s which will obviously not work:
By default RE0 will be Master and RE1 will be backup – but like the real MX you can change this via config and request them to failover and so on. Also “commit synchronize” is your friend 😉
set system commit synchronize set chassis redundancy routing-engine 0 master set chassis redundancy routing-engine 1 backup set chassis fpc 0 lite-mode set chassis fpc 1 lite-mode set chassis fpc 2 lite-mode set chassis fpc 3 lite-mode set chassis fpc 4 lite-mode set chassis fpc 5 lite-mode
Enjoy your “Powerhorse vMX” 😉
Hey chris,
I’m trying to setup a lab in which I’m using two vxm’s with dual RE’s and Dual FPC’s on each . I’m not able to ping one router to another. Can you help me on this .
TIA
Hi,
sure I can help you with that – can you send me your Setup and configs via chs[at]ip4[dot]de?
Regards
Chris
How were you able to resolve the ‘ping’ issue with multiple FPCs?
Anyway thanks, i found what was the issue.
how to use service pic’s in vmx17.1 ?
How do you implement the service PICs on a vMX?
when i login into Juniper site i found multiple image for downloading ( ESXi , KVM , limited , Nested ) and nothing match the eve site ,so which one do you recommended ?
You can use the KVM (qcow2) or the ESXi (and convert them) – both ways work fine
check this referrence
http://www.eve-ng.net/documentation/howto-s/109-howto-juniper-vmx-16-x-17-x
hi chris
i am trying setup vmx with dual RE (RE0 & RE1) and dual PFE (PFE0 & PFE1) i already follow your instruction create each directory RE0 & RE1 run fine, but not with VFP fpc0 & fpc1 not work. after i check pfe0 & pfe1 have same IP address 128.0.0.16
Did you use the files from the Juniper package? Every FPC (pfe) has a different image
Any way to get more than 8 FPC’s running? Past 8 they don’t seem to register correctly, and also tend to crash the RE when you try to reset/query/etc FPC8 and up.
Haven’t tried that
Seeing the same in my deployment.
I’ve logged a thread on Juniper vMX forum – https://forums.juniper.net/t5/vMX/vFPC-8-11-issue-on-vMX-18-2R1-9/m-p/465121#M1120
For me even a single VCP/VFP is failing. They boot up (After taking ages, even when giving them good resources (16gb ram, 8vcpus) .. but after all that time # show chassis fpc says that slot 0 is unresponsive. Thoughts?
I will try to setup the vMX in a similar environment as you described to check if this error is related to the images or the Hypervisor.
The Devices run fine on both my Boxes (ESX and Bare-Metal).
Hi Chris,
I’ve successfully deployed a vMX with 2 REs and two fpc, but when I tried to use it in a lab, i’ve noted two things:
1) The fpcX count one less interface than the standard fpc. For ex.: the interface labeled ge-x/0/1 by eve, is seen as ge-x/0/0 by junos.
2) I can’t ping between devices connected to different fpcs.
So i’ve tried to bridge all the fpcs ge-x/0/0 togeter. The system started to behave correctly. It seems that the ge-x/0/0 of the “non-slot-zero fpcs” are reserved as “intra-fpc-links”.
Did you experienced this?
Hi Luca,
I just connected all “int-Links” together and everything started to work just fine.
However, I did not test any connectivity at all with other Systems – just a test to see if I could run the vMX with multiple FPC’s.
i had the same issue
Just thought i’d point out, some versions of vMX seem to have a corrupt metadata-usb-fpc[x].img files (or at least, they dont work properly), but there is an easier alternative. Rather than using the fpc files, boot your vfpc, login as root.
mkdir -p /var/jnx/card/local
echo 1 > /var/jnx/card/local/slot
replace 1 with the fpc you want.
then “reboot”, i find this a little easier to do when working with eve myself
actually, to expand on that previous comment, how I do this is I create a VCP and VFP connected via a bridge, wait for them to boot up then from the vmx cli
start shell pfe network base-os fpc0 // this drops you to a root linux shell on fpc0
then
mkdir -p /var/jnx/card/local/
echo 1 > /var/jnx/card/local/slot
reboot
fpc0 now reboots into fpc1 and while thats happening i create another vfpc, connect it, boot it and it then boots into fpc0… do the same commands again (replacing 1 in the “echo” command with whichever slot you want the fpc to start as) and you can have multiple fpc’s. Reason I find this a little easier is I can use the same fpc image for any number of fpc’s rather than having an image per slot (which gets very crowded in the drop down if your using eve and have multiple vmx versions)
Hi Paul,
still you need to connect from ge-1/0/1 to use ge-1/0/0
did you find a solution for this?
Hi Chris, I followed your setup and everything comes up as planned. I can see both RE’s and my 5 fpc’s. routing engines gracefully switch over if I disconnect or power down the primary and everything seems to be working just fine. However, I connected this setup to alegacy vSRX, cant ping across. Connected it to a legacy vMX14.1, cant ping across. connected it to a C7200 iol image, cant ping across. Any suggestion?
Hi Alitaheri,
hard to tell from here – I could successfully connect anything lately when doing this again.
Could you share your setup? I will look into this.
BR
Chris
Hi Chris, Thank you for your response. To start with, I have 2 RE with 4fpc’s connected through a L2 bridge and everything comes up as far the MX goes. REs see the FPCs and I see the backup RE picking up once the master fails, yet I can not ping any device hooked to any of those FPCs. Im running vMX17.2 on a bare metal server. What other info do you need?
Hi Chris
Should one be able to perform hitless ISSU with this setup? Just for testing purposes.
Hi Casper,
I actually never tried that – but I like the Idea 😉
Theoretically it should work, but I don’t know if the vMX has a limitation for that.
BR
Chris
hi chris
I received a lot of help from your blog.
I’ve used the eve-ng tool to do various tests with vMX.
It consists of routing engine and two FPCs (Service-2G, vFPC) for NAT and Firewall test.
In VCP, vFPC did not connect after checking the normal connection and pinging the vPC.
Can you tell me what the problem is?
What Version do you run on your vMX?
I tested NAT and Firewall with vMS-MPC in vMX v18.1.
vFP (vMS-MPC = Service-2G / 4G / 10G module) has confirmed that the problem is not connected to the vPC.
The problem is that the vPC is connected by vCP (ge-0 / 0/1), so there is no link. This problem is presumed to be a bug in EVE-NG.
The solution was to set the value of vCP (ge-0 / 0/1) to ge-0 / 0/0 and connect it to the vPC.
We confirmed that the icmp packet and arp table are normally shared.
Eve-NG is v2.0.3-95.
Your blog has helped me a lot. Thank you very much.
Hi Chris,
What version do you running on your vMX can connect VCP to vMS-MPC(VFP)? kindly reply to me. Thanks bro so much.
Your blog is very interesting.
Hi Tuongpham,
I run a lot of Versions from 17.x up to 20.x
Any specific problem, that you run into?
Hi Chris,
What hardware do you running on your vMX 20.x can connect VCP to vMS-MPC?
I can’t run it for 18 vCPU 24G memory.
The vMX (single FPC) can run with 6GB in total, each vFP more adds another 4 GB
Hello. Do you know if it is possible to deploy multiple FPCs on Vmware ESXi ? Best regards
I have no idea and never tried that – but this sounds fun 🙂
In theory this should be possible – as long as FPC can talk to VCP
i need help on NAT and firewall features. How do we add vFP (vMS-MPC = Service-2G / 4G / 10G module) ? anyone can throw some light on this?
Hi, I tried using vMX 18.4R2.7 and deploy FPC0, FPC1, and VMS. Even I copy the meta data to the FPC folder, they use the same IP address 128.0.0.16/16 and failed to attach VCP. Any idea? Thanks.
Hi,
Anyone here can give more detail how to add vMS-MPC using EVE-NG?
Thanks and appreciate any help.
I found the link in eve-ng but the 18.2 images I obtained from Juniper only have 1 RE and 1 PIC. Any thoughts?
https://www.eve-ng.net/index.php/documentation/howtos/howto-add-juniper-vmx-16-x-17-x/
If you have the bundle (which is the file you need) there should be a subfolder containing the multiple metadata files (eg. metadata-usb-fpc0.img)
Hello Chris
I’m using vMX (20.3R1.8 ) and ran the process mention above, the REs start without any problem then apply the configuration to put one as master the other one as backup, also the commit synchronize. But I saw that the still appears as independently devices.
root@vMX-1-RE0> show task replication
Stateful Replication: Disabled
RE mode: Not Applicable
root@vMX-1-RE1# run show task replication
Stateful Replication: Disabled
RE mode: Not Applicable
I’m using the bridge and int (interfaces) to connect the both REs and also the FPCs, there is a command to see if they can see to each other ?
Regards
Wilson
“show chassis fpc” should show the other fpc’s if the correct metadata-file is used 🙂
Thanks Chris for the tutorial! I am using 18.2R1.9 with lite mode and I was able to get dual RE and multiple fpc setup without problem.
However, I noticed all the fpc port 9 is always down. (ge-0/0/9). I tried set chassis fpc 1 number-of-ports 12 but it didn’t help. Any thoughts?
Updating the number of ports on vfp fixed the issue. Thanks!
Hi Chris,
Did you ever manage to get VMX working with performance-mode in eve-ng ?
It seems to be missing the CPU from the FPC as below, nge091.ref for comparison of a working one.
root@NGE090.REF> show chassis hardware
Hardware inventory:
Item Version Part number Serial number Description
Chassis VM60FFB8DA7A VMX
Midplane
Routing Engine 0 RE-VMX
CB 0 VMX SCB
FPC 0 BUILTIN BUILTIN Virtual FPC
CPU
root@NGE091.REF> show chassis hardware
Hardware inventory:
Item Version Part number Serial number Description
Chassis VM60F6D4949C VMX
Midplane
Routing Engine 0 RE-VMX
CB 0 VMX SCB
FPC 0 BUILTIN BUILTIN Virtual FPC
CPU Rev. 1.0 RIOT-LITE BUILTIN
MIC 0 Virtual
PIC 0 BUILTIN BUILTIN Virtual
Hello Chris,
Great blog, and learning a lot on this, I do have a slight problem with my setup.
I have done the exact same thing as yourself exccept I am using a newer version 22.2R3.15
I have added 2 VFP’s and 2 VCP’s for now to test the functionallity, I have got both REs to work fine, however the 2 VFPs which in this case are fpc0 and fpc1 dont work together. I think I see the issue but I dont quite understand how it works.
when just turning on fpc0 I can see DHCP requests on the boot logs but they time out and I then presume it defaults to 128.0.0.16 is this normal behaviour? see below:
################################################################################
[ 77.453008] IPv6: ADDRCONF(NETDEV_UP): ext: link is not ready
Listening on LPF/ext/50:01:00:18:00:00
Sending on LPF/ext/50:01:00:18:00:00
Sending on Socket/fallback
DHCPDISCOVER on ext to 255.255.255.255 port 67 interval 6
DHCPDISCOVER on ext to 255.255.255.255 port 67 interval 12
DHCPDISCOVER on ext to 255.255.255.255 port 67 interval 11
DHCPDISCOVER on ext to 255.255.255.255 port 67 interval 13
DHCPDISCOVER on ext to 255.255.255.255 port 67 interval 7
DHCPDISCOVER on ext to 255.255.255.255 port 67 interval 12
No DHCPOFFERS received.
No working leases in persistent database – sleeping.
Ignoring unknown interface int=int.
done.
################################################################################
If this is normal then can you advise on this. When I then turn on fpc1 it also does the same but also gets the same IP and then both fpcs fail, which is understandable given it has the same IP address, I presumed it worked itself out and the metadata-usb-fpc0 and metadata-usb-fpc1 had different data in it to prevent duplicate assignment? I think this is what the issue I am having but unsure of the right way to fix it without breaking anything.
Thanks.