Don't swallow ImportError in temporary_directory() (#3037)

Fixes #3026

If the context wrapped by the temporary_directory() context manager
raised ImportError (for example because distutils.util cannot be
imported, #721 #1837), it would previously keep going, causing a
RuntimeError from contextlib:

    RuntimeError: generator didn't stop after throw()

Co-authored-by: Remi Rampin <r@remirampin.com>