IR: Drop metadata references more aggressively during teardown