Btrfs: fix hash overflow handling