Finally after weeks, no months of agonising failure though trial and error, I finally managed to get the outcome I desired with my Raspberry Pi 2!
A few years back I acquired a Cisco 3560 and quickly realised the potential of vlans and separate subnets for the purposes of testing among other valid reasons, and came to find that the nodes on most of the vlans could not communicate with the outside world (read: internet). It was then that I realised that something was wrong...
Long story short: the Netgear DGND4000 that I own does not route/NAT anything other than its resident subnet and I sure as heck was not going to implement double NAT!
Thanks be to LIbVirt's NAT networking which gave me an interim workaround and helped confirm this.
Getting the necessary bits
I then purchased a Raspberry Pi 2 along with a bunch of accessories. In the meantime (while I was waiting the excessively long shipping time). I did some research on the distributions that are capable of running on the bcm2709-based board and decided with OpenWRT. Yes, I know that I could have used Raspbian but OpenWRT seemed the most logical choice given the fact that it is essentially an internet router anyway, just without the wireless and ADSL modem.
Turns out I made the right choice despite the fact that OpenWRT is still in trunk (RC3 at the time of writing this).
Lastly (after destroying the extremely cheap Rpi2 case) I managed to get an image booted (helps when you use the bcm2709 not the bcm2708 barrier breaker version, thats for the Raspberry Model B!).
First of all, this would have gone a lot smother had I have just tested with the USB network adapter I bought along with the Pi, but it didn't get here in time with partial shipping.
I configured the switch with a trunk port with two vlans, one for the LAN side of things (internal link) and another for the WAN or pppoe (public/external/internets) and set the mode appropriately.
NOTE: VLANS and IP addresses have been altered so as to protect the actual configuration used in my network infrastructure. Call me paranoid.
Cisco 3650 partial configuration
ip address 192.168.69.1 255.255.255.248
! no interface defined for WAN because we do not want any L3 traffic
description Trunk port for Rpi2 VLAN's: 20, 69
switchport access vlan 69
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 20,69
switchport mode trunk
description Link to DM111Pv2 modem (bridged) for PPPoE/L2 traffic
switchport access vlan 20
ip route 0.0.0.0 0.0.0.0 192.168.69.66
OpenWRT network configuration
root@OpenWRT# vi /etc/config/network
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config interface 'lan'
option proto 'static'
option delegate '0'
option _orig_ifname 'eth0'
option _orig_bridge 'false'
option ifname 'eth0.69'
option ipaddr '192.168.69.66'
option netmask '255.255.255.248'
option interface 'lan'
option target '192.168.0.0'
option netmask '255.255.0.0'
option gateway '192.168.69.1'
config interface 'WAN'
option proto 'pppoe'
option ifname 'eth0.20'
option delegate '0'
option username 'myusername'
option password 'mY$eCr3tP4sSw0rD'
By now you may be wondering, "Why is there no IP addresses or switch virtual interface for vlan 20"? There is no need for it! That, and the fact one might only want traffic to go via one vlan and then the other (remember, this is essentially a router on a stick implementation and we want to separate the vlan's into L3 traffic for one and L2 for the other per requirements).
If you were thinking: "The netmask and destination network IP for the LAN route is wrong!", you would be incorrect. This is a perfectly legitimate summary route. It allows for much easier (read: slack) administration so one does not have to manage multiple static routes for subnets added or removed from the network (short of running a routing protocol) and it has the added benefit of consuming less memory and is a much more flexible approach for this design. Neat huh? I thought so too :-)
Let it be said that although this configuration is very simple, there where many hurdles accompanied by many choice words along the way. The one single most important thing that I kept getting wrong was routing. I had to remember to change the 'gateway of last resort' (Cisco's way of saying default route) on the switch so that all the subnets will route to the internet and the static (summarised) route for traffic to get back into the network from whence they came. That and trying to test this when the internet is depended upon so much by the two people in this household, was frustrating as my change windows where often short and had to be rolled back constantly.
Lastly, I must say that "out-of-the-box" pppoe/nat/routing on OpenWRT worked with like a charm with minimal configuration, however I will need to develop the scenario a little further so I can secure the connection by way of its firewall (read: iptables), but that itself is a beast I have yet to conquer.