Let $\displaystyle \vec{a} = x\hat{i}+y\hat{j}+z\hat{k}$ and $\displaystyle \vec{b} = \sqrt{7}\hat{i}+\sqrt{11}\hat{j}+3\sqrt{2}\hat{k}$
Using $\bigg|\vec{a}\times \vec{b}\bigg|^2=|\vec{a}|^2|\vec{b}|^2-\bigg(\vec{a}\cdot \vec{b}\bigg)\leq |\vec{a}|^2|\vec{b}|^2$
So
$(3\sqrt{2}y-\sqrt{11}z)^2+(\sqrt{7}z-3\sqrt{2}x)^2+(\sqrt{11}x-\sqrt{7}y)^2\leq 36$
Equality hold when $\sqrt{7}x+\sqrt{11}y+3\sqrt{z}=0.$