The feared grains of salt have reared their ugly heads: the growth exponent 1/186 that I mentioned in my last post has been reduced to 1/744. I had missed the fact that, in order to avoid dealing with elements of trace 0, I first had to replace the generating set $latex H$ with $latex H\cdot H$, alas… But the previous exponent does work for any generating set which contains a regular semisimple element with non-zero trace.
[P.S. For the Lubotzky group, the spectral gap goes down to something like $latex 2^{-2^{36}}$…]