Gradual rollout of the permission change
The change reached ClickHouse nodes in stages. Each five-minute cycle could run on an updated node or an old one. The file was good or bad depending on that node. Once all nodes were updated, every file was bad and the failure became stable.