Fix constness problems now that the cast operators preserve the constness