The issue has been found and addresses in RHEL6.5 in the ixgbe driver.
Since I wasn't getting anywhere working with Intel, I brought it to the attention of Redhat. They
worked in conjunction with HP (it was their device) and the between the two of them,
the oncovered and solved the issue.
I cannot tell you what specifically it was, but they updated the ixgbe driver (3.15.1-k) and
rolled that out in the new RHEL6.5 kernel.
I suppose it will eventually rolled out back into the base intel driver. As I stated earlier, the
multicast register was getting cleared when setting MTU size greater than 1500
bytes && SR-IOV interfaces were used.
Thanks,
Shawn