Don't swallow ImportError in temporary_directory()

Fixes #3026

If the context wrapped by the temporary_directory() context manager
raised ImportError (for example because distutils.util cannot be
imported, #721 #1837), it would previously keep going, causing a
RuntimeError from contextlib:

    RuntimeError: generator didn't stop after throw()