Take alignment into account in isSafeToLoadUnconditionally