They make a special voice coil cement for this purpose. It is an awesome product, takes a while to set up but it won't ever fail on you. There is one hitch: this is strictly a commercial product, not available in stores, and I've only seen it in quart cans. If you have someone in your area who specializes in re-coning speakers maybe you can obtain a jigger-full from them.

The best way I've found to handle voice coil cement is with something similar to a wooden matchstick, a drop at a time. You will be amazed at the consistency, strength, and flexibility of this product. It is also useful for other purposes such as sealing electronic components, like when you have to fish a lead from a transformer, etc. It can even be used in a pinch to insulate small areas of electronic wiring. Once in place it's there to stay.