There is an old metal-working trick which might help you here, and it's less invasive than glue: grinding dust.
Sprinkle a little but of the finest/strongest metal dust you have in the hole. (And I'm sure you've got lots of it from making picks, right?)
They put the proper-sized wrench in. The dust is non-compressible for the most part, and tend to fill in holes and stick well.