[CodeGen] Re-apply r234809 (concat of scalars), with an x86_mmx fix.