If when you try to permanently delete the media and you are shown an "internal server error" message like the one in the picture bellow, please follow the steps described here.
Make sure that the media you are trying to delete is not linked to a post.
It the media is linked to a post, you will have to restore the file.
Delete the posts and after that delete the media as well.