When trying to upload many files (>250) to a private bucket, the hf sync utility sometimes freezes completely. Is this a known issue?
250 files does not sound like enough to freeze sync by itself.
I would first check if it hangs before upload or during upload. Run the same sync once with --dry-run and once with --verbose. If dry-run is fine but the real upload stops, then it is probably not the file count. More likely one file, one path, or one transfer is getting stuck.
Also check if it always stops around the same file. If yes, that is the useful clue. If it stops at random places, I would split the folder and upload by smaller chunks just to narrow it down. Not pretty, but it usually tells you fast whether the problem is the CLI or some object in the folder.
Would also help to post the huggingface_hub version and the exact command. Otherwise this is hard to tell apart from a CLI bug, a network stall, or one weird file.
I’m running hf sync on a bucket with a few hundred thousand files. I’m using version 1.14.0, and from my experience, the issue seems to have become worse in newer versions.
I even wrote a custom script that downloads files in batches of 200, but the utility still freezes from time to time. It seems that all those issues are connected to hidden limit on the number of files that can be transferred per minute. To work around that, you may need to package the data into a format like WebDataset.
Not same but maybe related issue: OOM Killed during bucket upload · Issue #4145 · huggingface/huggingface_hub · GitHub
Also when uploading large (or large number of) files: Uploads with `New Data Upload` over 2GB get stuck in Windows 11 with Python 3.12 · Issue #3871 · huggingface/huggingface_hub · GitHub