{ "cells": [ { "cell_type": "markdown", "id": "34582602", "metadata": {}, "source": [ "# Master Notebook for Hugging Face Uploads and Downloads — Robust V22\n", "\n", "Built and tested for `huggingface_hub 1.24.0` (released July 17, 2026) with its built-in `hf_xet` transfer engine. Requires Python 3.10 or newer.\n", "\n", "**This revision does not inspect, reopen, or depend on the notebook filename.** You can rename, duplicate, move, or upload the notebook under any `.ipynb` name.\n", "\n", "Only edit values inside the clearly marked `EDIT ONLY THESE VALUES` blocks. Run the other cells as-is, from top to bottom. Interrupted transfers can be resumed by running the same transfer cell again.\n", "\n", "Windows paths: use a raw string such as `r\"D:\\Models\\folder\"`. Linux paths: use a normal path such as `\"/home/Ubuntu/models\"`.\n", "\n", "Latest notebook post: https://www.patreon.com/posts/104672510" ] }, { "cell_type": "markdown", "id": "a7ae61a4", "metadata": {}, "source": [ "## 1. Required setup — run before every upload or download\n", "\n", "Run the next cell once after starting or restarting the kernel. Wait until **TRANSFER ENGINE READY** is printed.\n", "\n", "The cell installs the tested Hub version and `ipywidgets` into the active kernel environment, enables high-performance Xet before importing it, validates the loaded versions, and defines all transfer helpers directly. It does not search for another cell or read this notebook from disk." ] }, { "cell_type": "code", "execution_count": null, "id": "2fba6c7e", "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true, "source_hidden": true }, "tags": [ "required-setup", "hide-input", "hf-transfer-engine-v22" ] }, "outputs": [], "source": [ "%pip install -q -U \"huggingface_hub==1.24.0\" \"ipywidgets>=8.1,<9\"\n", "\n", "import html as _html\n", "import itertools as _itertools\n", "import logging as _python_logging\n", "import os as _os\n", "import sys as _sys\n", "import threading as _threading\n", "import time as _time\n", "import uuid as _uuid\n", "import warnings as _warnings\n", "from collections import deque as _deque\n", "from contextlib import contextmanager as _contextmanager\n", "from importlib import import_module as _import_module\n", "from importlib.metadata import PackageNotFoundError as _PackageNotFoundError\n", "from importlib.metadata import version as _package_version\n", "from pathlib import Path as _Path\n", "from pathlib import PurePosixPath as _PurePosixPath\n", "from urllib.parse import quote as _url_quote\n", "\n", "# These values must be set before importing huggingface_hub/hf_xet.\n", "for _legacy_or_conflicting_variable in (\n", " \"HF_HUB_ENABLE_HF_TRANSFER\",\n", " \"HF_HUB_DISABLE_PROGRESS_BARS\",\n", " \"HF_HUB_DISABLE_XET\",\n", " \"HF_DEBUG\",\n", "):\n", " _os.environ.pop(_legacy_or_conflicting_variable, None)\n", "_os.environ[\"HF_XET_HIGH_PERFORMANCE\"] = \"1\"\n", "_os.environ[\"HF_HUB_VERBOSITY\"] = \"warning\"\n", "_os.environ.setdefault(\"HF_HUB_DOWNLOAD_TIMEOUT\", \"60\")\n", "_os.environ.setdefault(\"HF_HUB_ETAG_TIMEOUT\", \"30\")\n", "\n", "_HUB_WAS_ALREADY_IMPORTED = \"huggingface_hub\" in _sys.modules\n", "\n", "from tqdm.std import TqdmWarning as _TqdmWarning\n", "from tqdm.std import tqdm as _BaseTqdm\n", "\n", "# tqdm.auto warns when optional Jupyter widgets are absent, even though this notebook\n", "# deliberately uses the widget-free display renderer below. Preload its standard\n", "# fallback while suppressing only that harmless, well-known warning.\n", "with _warnings.catch_warnings():\n", " _warnings.filterwarnings(\n", " \"ignore\",\n", " message=r\"IProgress not found\\..*\",\n", " category=_TqdmWarning,\n", " )\n", " _import_module(\"tqdm.auto\")\n", " import huggingface_hub as _hub\n", " from huggingface_hub import HfApi as _HfApi\n", " from huggingface_hub import constants as _hub_constants\n", " from huggingface_hub import snapshot_download as _snapshot_download\n", " from huggingface_hub.utils import enable_progress_bars as _enable_progress_bars\n", " from huggingface_hub.utils import validate_repo_id as _validate_repo_id\n", "from IPython import get_ipython as _get_ipython\n", "from IPython.display import HTML as _HTML\n", "from IPython.display import display as _display\n", "from IPython.display import update_display as _update_display\n", "from packaging.specifiers import SpecifierSet as _SpecifierSet\n", "from packaging.version import Version as _Version\n", "\n", "_EXPECTED_HUB_VERSION = _Version(\"1.24.0\")\n", "_SUPPORTED_XET_VERSIONS = _SpecifierSet(\">=1.5.1,<2.0.0\")\n", "\n", "if _Version(_hub.__version__) != _EXPECTED_HUB_VERSION:\n", " if _HUB_WAS_ALREADY_IMPORTED:\n", " raise RuntimeError(\n", " f\"This kernel already loaded huggingface_hub {_hub.__version__}. \"\n", " f\"Restart the kernel, then run setup again to load {_EXPECTED_HUB_VERSION}.\"\n", " )\n", " raise RuntimeError(\n", " f\"Expected huggingface_hub {_EXPECTED_HUB_VERSION}, but imported {_hub.__version__}. \"\n", " \"Check the installation output above, restart the kernel, and run setup again.\"\n", " )\n", "\n", "try:\n", " _XET_VERSION_TEXT = _package_version(\"hf-xet\")\n", "except _PackageNotFoundError as _error:\n", " raise RuntimeError(\n", " \"hf-xet is not installed for this Python/platform. Install a supported 64-bit Python build, \"\n", " \"restart the kernel, and run setup again.\"\n", " ) from _error\n", "\n", "if _Version(_XET_VERSION_TEXT) not in _SUPPORTED_XET_VERSIONS:\n", " raise RuntimeError(\n", " f\"Unsupported hf-xet {_XET_VERSION_TEXT}; expected {_SUPPORTED_XET_VERSIONS}. \"\n", " \"Restart the kernel after setup installs compatible dependencies.\"\n", " )\n", "\n", "# Import hf_xet only after high-performance mode is configured.\n", "try:\n", " _import_module(\"hf_xet\")\n", "except Exception as _error:\n", " raise RuntimeError(f\"hf-xet could not be loaded: {_error}\") from _error\n", "\n", "if getattr(_hub_constants, \"HF_HUB_DISABLE_XET\", False):\n", " raise RuntimeError(\n", " \"Xet was disabled before setup ran. Restart the kernel, then run this notebook from the top.\"\n", " )\n", "if not getattr(_hub_constants, \"HF_XET_HIGH_PERFORMANCE\", False):\n", " raise RuntimeError(\n", " \"High-performance Xet mode was not enabled before huggingface_hub loaded. \"\n", " \"Restart the kernel, then run this notebook from the top.\"\n", " )\n", "\n", "_enable_progress_bars()\n", "_python_logging.getLogger(\"huggingface_hub\").setLevel(_python_logging.WARNING)\n", "_python_logging.getLogger(\"httpx\").setLevel(_python_logging.WARNING)\n", "_python_logging.getLogger(\"httpcore\").setLevel(_python_logging.WARNING)\n", "\n", "_VALID_REPO_TYPES = {\"model\", \"dataset\", \"space\"}\n", "_REFRESH_SECONDS = 1.0\n", "_SPEED_WINDOW_SECONDS = 2.0\n", "_PROGRESS_PATCH_LOCK = _threading.RLock()\n", "\n", "try:\n", " _upload_pipeline = _import_module(\"huggingface_hub._upload_pipeline\")\n", "except Exception:\n", " _upload_pipeline = None\n", "\n", "\n", "def _is_notebook():\n", " \"\"\"Return True for Jupyter, JupyterLab, VS Code notebooks, and Colab.\"\"\"\n", " shell = _get_ipython()\n", " if shell is None:\n", " return False\n", " shell_name = shell.__class__.__name__\n", " return (\n", " shell_name in {\"ZMQInteractiveShell\", \"Shell\"}\n", " or \"ipykernel\" in _sys.modules\n", " or \"google.colab\" in _sys.modules\n", " )\n", "\n", "\n", "def _format_size(value):\n", " value = float(max(0, value))\n", " for unit in (\"B\", \"KB\", \"MB\", \"GB\", \"TB\", \"PB\"):\n", " if value < 1000 or unit == \"PB\":\n", " return f\"{value:.0f} {unit}\" if value >= 100 else f\"{value:.1f} {unit}\"\n", " value /= 1000\n", "\n", "\n", "def _format_duration(seconds):\n", " seconds = int(max(0, seconds))\n", " hours, seconds = divmod(seconds, 3600)\n", " minutes, seconds = divmod(seconds, 60)\n", " if hours:\n", " return f\"{hours}h {minutes}m {seconds}s\"\n", " if minutes:\n", " return f\"{minutes}m {seconds}s\"\n", " return f\"{seconds}s\"\n", "\n", "\n", "def _bar(current, total, width=20):\n", " if total <= 0:\n", " return \".\" * width\n", " filled = int(min(max(current / total, 0), 1) * width)\n", " return \"#\" * filled + \".\" * (width - filled)\n", "\n", "\n", "class _StatusArea:\n", " \"\"\"One updateable output area with a plain-text fallback.\"\"\"\n", "\n", " def __init__(self, title):\n", " self.title = str(title)\n", " self._lines = {}\n", " self._order = []\n", " self._handle = None\n", " self._display_id = f\"hf-transfer-{_uuid.uuid4().hex}\"\n", " self._last_publish = 0.0\n", " self._last_fallback = 0.0\n", " self._lock = _threading.RLock()\n", "\n", " def set_line(self, key, text, force=False):\n", " self.set_lines([(key, text)], force=force)\n", "\n", " def set_lines(self, lines, force=False):\n", " with self._lock:\n", " for key, text in lines:\n", " if key not in self._lines:\n", " self._order.append(key)\n", " self._lines[key] = str(text).replace(\"\\r\", \"\").replace(\"\\n\", \" \")\n", " self._publish_if_due(force)\n", "\n", " def finish(self):\n", " with self._lock:\n", " self._publish_if_due(True)\n", "\n", " def _payload(self):\n", " text = self.title\n", " if self._order:\n", " text += \"\\n\" + \"\\n\".join(self._lines[key] for key in self._order)\n", " return text, _HTML(\n", " \"
\"\n",
    "            + _html.escape(text)\n",
    "            + \"
\"\n", " )\n", "\n", " def _publish_if_due(self, force):\n", " now = _time.monotonic()\n", " if not force and now - self._last_publish < _REFRESH_SECONDS:\n", " return\n", " self._last_publish = now\n", " text, payload = self._payload()\n", "\n", " try:\n", " if self._handle is None:\n", " self._handle = _display(payload, display_id=self._display_id)\n", " elif callable(getattr(self._handle, \"update\", None)):\n", " self._handle.update(payload)\n", " else:\n", " _update_display(payload, display_id=self._display_id)\n", " return\n", " except Exception:\n", " # Some frontends do not implement display IDs; use a throttled text fallback.\n", " pass\n", "\n", " if force or now - self._last_fallback >= 10:\n", " print(text.replace(\"\\n\", \" | \"), flush=True)\n", " self._last_fallback = now\n", "\n", "\n", "_tqdm_ids = _itertools.count()\n", "\n", "\n", "class _NotebookTqdm(_BaseTqdm):\n", " _area = None\n", "\n", " def __init__(self, *args, **kwargs):\n", " self._status_key = f\"bar-{next(_tqdm_ids)}\"\n", " self._uses_measured_speed = False\n", " kwargs.setdefault(\"ascii\", True)\n", " kwargs.setdefault(\"mininterval\", _REFRESH_SECONDS)\n", " kwargs.setdefault(\"maxinterval\", _REFRESH_SECONDS)\n", " kwargs.setdefault(\"miniters\", 1)\n", " kwargs.setdefault(\"dynamic_ncols\", False)\n", " kwargs.setdefault(\"ncols\", 104)\n", " super().__init__(*args, **kwargs)\n", " self._measured_rate = 0.0\n", " self._rate_samples = _deque([(_time.monotonic(), float(self.n))])\n", "\n", " def update(self, n=1):\n", " now = _time.monotonic()\n", " reset_after_idle = (\n", " self._rate_samples\n", " and now - self._rate_samples[-1][0] > _SPEED_WINDOW_SECONDS\n", " and float(self.n) == self._rate_samples[-1][1]\n", " )\n", " result = super().update(n)\n", " if not self.disable:\n", " now = _time.monotonic()\n", " if reset_after_idle:\n", " # The first burst has no reliable start time; measure from the next update.\n", " self._rate_samples.clear()\n", " self._rate_samples.append((now, float(self.n)))\n", " self._measured_rate = 0.0\n", " return result\n", " self._rate_samples.append((now, float(self.n)))\n", " cutoff = now - _SPEED_WINDOW_SECONDS\n", " while len(self._rate_samples) > 2 and self._rate_samples[1][0] <= cutoff:\n", " self._rate_samples.popleft()\n", " started, initial = self._rate_samples[0]\n", " elapsed = now - started\n", " if elapsed > 0:\n", " self._measured_rate = max(0.0, (float(self.n) - initial) / elapsed)\n", " return result\n", "\n", " def set_postfix_str(self, s=\"\", refresh=True):\n", " # Xet supplies deliberately conservative warm-up rates. Replace only its\n", " # byte-per-second postfix with this bar's direct rolling measurement.\n", " if str(s).strip().endswith(\"B/s\") and hasattr(self, \"_measured_rate\"):\n", " self._uses_measured_speed = True\n", " s = f\"{self.format_sizeof(self._measured_rate)}B/s \".rjust(10, \" \")\n", " return super().set_postfix_str(s=s, refresh=refresh)\n", "\n", " def display(self, msg=None, pos=None):\n", " area = type(self)._area\n", " if area is None:\n", " return super().display(msg=msg, pos=pos)\n", " area.set_line(self._status_key, msg if msg is not None else str(self))\n", "\n", " def close(self):\n", " if self.disable:\n", " return\n", " if self._uses_measured_speed:\n", " self.set_postfix_str(\"0B/s\", refresh=False)\n", " area = type(self)._area\n", " super().close()\n", " if area is not None:\n", " area.set_line(self._status_key, str(self), force=True)\n", "\n", "\n", "@_contextmanager\n", "def _notebook_tqdm_area(title):\n", " if not _is_notebook():\n", " yield None\n", " return\n", " area = _StatusArea(title)\n", " previous = _NotebookTqdm._area\n", " _NotebookTqdm._area = area\n", " try:\n", " yield _NotebookTqdm\n", " finally:\n", " area.finish()\n", " _NotebookTqdm._area = previous\n", "\n", "\n", "_UPLOAD_DISPLAY_BASE = getattr(_upload_pipeline, \"_LiveDisplay\", None) if _upload_pipeline is not None else None\n", "_UPLOAD_DISPLAY_METHODS = {\n", " \"_line_preparing\",\n", " \"_line_committing\",\n", " \"new_xet_callback\",\n", " \"notify_xet_uploaded\",\n", "}\n", "_UPLOAD_DISPLAY_COMPATIBLE = bool(\n", " _UPLOAD_DISPLAY_BASE\n", " and all(callable(getattr(_UPLOAD_DISPLAY_BASE, name, None)) for name in _UPLOAD_DISPLAY_METHODS)\n", ")\n", "\n", "if _UPLOAD_DISPLAY_COMPATIBLE:\n", "\n", " class _NotebookUploadDisplay(_UPLOAD_DISPLAY_BASE):\n", " \"\"\"Notebook renderer with independently measured logical and network rates.\"\"\"\n", "\n", " _area = None\n", "\n", " def __init__(self, total_files, enabled=True):\n", " super().__init__(total_files=total_files, enabled=enabled)\n", " self._active = bool(enabled)\n", " self._tty = False\n", " self._logical_bytes = 0\n", " self._logical_rate = 0.0\n", " self._network_rate = 0.0\n", " self._last_active_logical_rate = 0.0\n", " self._last_active_network_rate = 0.0\n", " self._speed_samples = _deque([(_time.monotonic(), 0, 0)])\n", "\n", " def new_xet_callback(self):\n", " if not self._active:\n", " return None\n", " previous_logical = 0\n", " previous_transfer = 0\n", "\n", " def callback(group_report, item_reports):\n", " nonlocal previous_logical, previous_transfer\n", " logical = max(0, int(getattr(group_report, \"total_bytes_completed\", 0) or 0))\n", " transfer = max(0, int(getattr(group_report, \"total_transfer_bytes_completed\", 0) or 0))\n", " with self._lock:\n", " self._logical_bytes += max(0, logical - previous_logical)\n", " self._xet_bytes += max(0, transfer - previous_transfer)\n", " previous_logical = logical\n", " previous_transfer = transfer\n", " for item in (item_reports or {}).values():\n", " if item.total_bytes > 0 and item.bytes_completed == item.total_bytes:\n", " self._xet_done.add(item.item_name)\n", " self._record_speed_sample(_time.monotonic())\n", "\n", " return callback\n", "\n", " def _record_speed_sample(self, now):\n", " self._speed_samples.append((now, self._logical_bytes, self._xet_bytes))\n", " cutoff = now - _SPEED_WINDOW_SECONDS\n", " while len(self._speed_samples) > 2 and self._speed_samples[1][0] <= cutoff:\n", " self._speed_samples.popleft()\n", "\n", " def _update_measured_rates(self, now=None):\n", " now = _time.monotonic() if now is None else float(now)\n", " self._record_speed_sample(now)\n", " started, logical, transfer = self._speed_samples[0]\n", " elapsed = now - started\n", " if elapsed <= 0:\n", " self._logical_rate = 0.0\n", " self._network_rate = 0.0\n", " return\n", " self._logical_rate = max(0.0, (self._logical_bytes - logical) / elapsed)\n", " self._network_rate = max(0.0, (self._xet_bytes - transfer) / elapsed)\n", " if self._logical_rate > 0:\n", " self._last_active_logical_rate = self._logical_rate\n", " if self._network_rate > 0:\n", " self._last_active_network_rate = self._network_rate\n", "\n", " def _line_uploading(self):\n", " if self._xet_total == 0:\n", " bar = _bar(1, 1) if self._prepared >= self._total else _bar(0, 1)\n", " return f\" Uploading {bar} -\"\n", " completed = len(self._xet_done)\n", " done = \" complete\" if self._prepared >= self._total and completed >= self._xet_total else \"\"\n", " return (\n", " f\" Uploading {_bar(completed, self._xet_total)} \"\n", " f\"{completed:,} / {self._xet_total:,} files{done}\"\n", " )\n", "\n", " def _line_throughput(self):\n", " if self._logical_bytes == 0 and self._xet_bytes == 0:\n", " return \" Throughput measuring...\"\n", " finished = self._xet_total > 0 and len(self._xet_done) >= self._xet_total\n", " logical_rate = self._last_active_logical_rate if finished else self._logical_rate\n", " network_rate = self._last_active_network_rate if finished else self._network_rate\n", " qualifier = \"last active\" if finished else f\"rolling {_SPEED_WINDOW_SECONDS:.0f}s\"\n", " return (\n", " f\" Throughput effective {_format_size(logical_rate)}/s | \"\n", " f\"network {_format_size(network_rate)}/s ({qualifier})\"\n", " )\n", "\n", " def _line_data(self):\n", " return (\n", " f\" Xet data {_format_size(self._logical_bytes)} processed | \"\n", " f\"{_format_size(self._xet_bytes)} sent after deduplication\"\n", " )\n", "\n", " def start(self):\n", " if not self._active:\n", " return\n", " self._publish(True)\n", " self._thread = _threading.Thread(\n", " target=self._render_loop,\n", " name=\"hf-notebook-upload-progress\",\n", " daemon=True,\n", " )\n", " self._thread.start()\n", "\n", " def close(self):\n", " if self._thread is not None:\n", " self._stop_event.set()\n", " self._thread.join()\n", " self._publish(True)\n", "\n", " def _render_loop(self):\n", " while not self._stop_event.wait(_REFRESH_SECONDS):\n", " self._publish(False)\n", "\n", " def _publish(self, force):\n", " area = type(self)._area\n", " if area is None:\n", " return\n", " with self._lock:\n", " self._update_measured_rates()\n", " lines = [\n", " (\"prepare\", self._line_preparing()),\n", " (\"upload\", self._line_uploading()),\n", " (\"throughput\", self._line_throughput()),\n", " (\"data\", self._line_data()),\n", " (\"commit\", self._line_committing()),\n", " ]\n", " area.set_lines(lines, force=force)\n", "\n", "else:\n", " _NotebookUploadDisplay = None\n", "\n", "\n", "@_contextmanager\n", "def _folder_upload_progress():\n", " if not _is_notebook() or _NotebookUploadDisplay is None or _upload_pipeline is None:\n", " yield\n", " return\n", "\n", " with _PROGRESS_PATCH_LOCK:\n", " area = _StatusArea(\"UPLOAD PROGRESS\")\n", " previous_class = _upload_pipeline._LiveDisplay\n", " previous_area = _NotebookUploadDisplay._area\n", " _NotebookUploadDisplay._area = area\n", " _upload_pipeline._LiveDisplay = _NotebookUploadDisplay\n", " try:\n", " yield\n", " finally:\n", " _upload_pipeline._LiveDisplay = previous_class\n", " _NotebookUploadDisplay._area = previous_area\n", " area.finish()\n", "\n", "\n", "@_contextmanager\n", "def _single_upload_progress():\n", " if not _is_notebook():\n", " yield\n", " return\n", "\n", " try:\n", " reporter = _import_module(\"huggingface_hub.utils._xet_progress_reporting\")\n", " except Exception:\n", " yield\n", " return\n", "\n", " with _PROGRESS_PATCH_LOCK:\n", " previous_tqdm = reporter.tqdm\n", " with _notebook_tqdm_area(\"UPLOAD PROGRESS\") as tqdm_class:\n", " reporter.tqdm = tqdm_class\n", " try:\n", " yield\n", " finally:\n", " reporter.tqdm = previous_tqdm\n", "\n", "\n", "def _repo_type(value):\n", " value = str(value).strip().lower()\n", " if value not in _VALID_REPO_TYPES:\n", " raise ValueError(f\"REPO_TYPE must be one of: {', '.join(sorted(_VALID_REPO_TYPES))}\")\n", " return value\n", "\n", "\n", "def _repo_id(value, upload=False):\n", " value = str(value).strip().strip(\"/\")\n", " if not value or value.casefold().startswith((\"yourusername/\", \"username/\")):\n", " raise ValueError(\"Set REPO_ID in the EDIT ONLY THESE VALUES block.\")\n", " try:\n", " _validate_repo_id(value)\n", " except Exception as error:\n", " raise ValueError(f\"Invalid REPO_ID {value!r}: {error}\") from error\n", " if upload and \"/\" not in value:\n", " raise ValueError(\"Upload REPO_ID must be 'username/repo-name' or 'organization/repo-name'.\")\n", " return value\n", "\n", "\n", "def _repo_url(repo_id, repo_type):\n", " prefix = {\"model\": \"\", \"dataset\": \"datasets/\", \"space\": \"spaces/\"}[repo_type]\n", " return f\"https://huggingface.co/{prefix}{repo_id}\"\n", "\n", "\n", "def _local_path(value, label):\n", " text = _os.path.expandvars(str(value)).strip()\n", " if not text:\n", " raise ValueError(f\"Set {label} in the EDIT ONLY THESE VALUES block.\")\n", " normalized = text.replace(\"\\\\\", \"/\").casefold()\n", " if normalized.startswith((\"/path/to/\", \"path/to/\", \"c:/path/to/\", \"d:/path/to/\")):\n", " raise ValueError(f\"Replace the placeholder {label} with a real local path.\")\n", " path = _Path(text).expanduser()\n", " if not path.is_absolute():\n", " path = _Path.cwd() / path\n", " return path.resolve(strict=False)\n", "\n", "\n", "def _remote_path(value, label, *, allow_blank=False):\n", " value = str(value).strip().replace(\"\\\\\", \"/\")\n", " if not value:\n", " if allow_blank:\n", " return \"\"\n", " raise ValueError(f\"Set {label} in the EDIT ONLY THESE VALUES block.\")\n", " if value.startswith(\"/\"):\n", " raise ValueError(f\"{label} must be relative to the repository root (no leading slash).\")\n", " value = value.rstrip(\"/\")\n", " path = _PurePosixPath(value)\n", " if not path.parts or any(part in {\"\", \".\", \"..\"} for part in path.parts):\n", " raise ValueError(f\"{label} contains an invalid path segment: {value!r}\")\n", " if \"\\x00\" in value:\n", " raise ValueError(f\"{label} contains a null byte.\")\n", " return path.as_posix()\n", "\n", "\n", "def _run_transfer(kind, source, target, action):\n", " print(f\"{kind} STARTED\")\n", " print(f\"From: {source}\")\n", " print(f\"To: {target}\")\n", " started = _time.monotonic()\n", " try:\n", " result = action()\n", " except KeyboardInterrupt:\n", " print(f\"{kind} INTERRUPTED - run this cell again to resume.\")\n", " raise\n", " except Exception as error:\n", " print(f\"{kind} FAILED - {type(error).__name__}: {error}\")\n", " print(\"The Hub/Xet clients retry transient chunks internally. After fixing the reported cause, run this cell again.\")\n", " raise\n", " print(f\"{kind} COMPLETED in {_format_duration(_time.monotonic() - started)}\")\n", " return result\n", "\n", "\n", "def _print_commit(result):\n", " commit_url = getattr(result, \"commit_url\", None)\n", " if commit_url:\n", " print(f\"Commit: {commit_url}\")\n", "\n", "\n", "def upload_folder_fast(repo_id, folder_path, repo_type=\"model\"):\n", " repo_id = _repo_id(repo_id, upload=True)\n", " repo_type = _repo_type(repo_type)\n", " folder = _local_path(folder_path, \"FOLDER_PATH\")\n", " if not folder.is_dir():\n", " raise FileNotFoundError(f\"Upload folder does not exist or is not a directory: {folder}\")\n", " api = _HfApi()\n", " target = _repo_url(repo_id, repo_type)\n", "\n", " def action():\n", " api.create_repo(repo_id=repo_id, repo_type=repo_type, exist_ok=True)\n", " with _folder_upload_progress():\n", " return api.upload_folder(\n", " repo_id=repo_id,\n", " repo_type=repo_type,\n", " folder_path=folder,\n", " commit_message=f\"Upload folder with huggingface_hub {_hub.__version__}\",\n", " )\n", "\n", " result = _run_transfer(\"UPLOAD\", folder, target, action)\n", " _print_commit(result)\n", " return result\n", "\n", "\n", "def upload_file_fast(repo_id, local_file, remote_file=None, repo_type=\"model\"):\n", " repo_id = _repo_id(repo_id, upload=True)\n", " repo_type = _repo_type(repo_type)\n", " local_file = _local_path(local_file, \"LOCAL_FILE\")\n", " if not local_file.is_file():\n", " raise FileNotFoundError(f\"Upload file does not exist or is not a file: {local_file}\")\n", " remote_file = _remote_path(remote_file, \"REMOTE_FILE\", allow_blank=True) if remote_file is not None else \"\"\n", " if not remote_file:\n", " remote_file = local_file.name\n", " api = _HfApi()\n", " target = f\"{_repo_url(repo_id, repo_type)}/blob/main/{_url_quote(remote_file, safe='/')}\"\n", "\n", " def action():\n", " api.create_repo(repo_id=repo_id, repo_type=repo_type, exist_ok=True)\n", " with _single_upload_progress():\n", " return api.upload_file(\n", " repo_id=repo_id,\n", " repo_type=repo_type,\n", " path_or_fileobj=local_file,\n", " path_in_repo=remote_file,\n", " commit_message=f\"Upload {remote_file}\",\n", " )\n", "\n", " result = _run_transfer(\"UPLOAD\", local_file, target, action)\n", " _print_commit(result)\n", " return result\n", "\n", "\n", "def _download_snapshot(repo_id, local_dir, repo_type, allow_patterns=None, validator=None):\n", " repo_id = _repo_id(repo_id)\n", " repo_type = _repo_type(repo_type)\n", " local_dir = _local_path(local_dir, \"LOCAL_DIR\")\n", " if local_dir.exists() and not local_dir.is_dir():\n", " raise NotADirectoryError(f\"LOCAL_DIR exists but is not a directory: {local_dir}\")\n", " local_dir.mkdir(parents=True, exist_ok=True)\n", " source_url = _repo_url(repo_id, repo_type)\n", "\n", " def action():\n", " kwargs = dict(\n", " repo_id=repo_id,\n", " repo_type=repo_type,\n", " local_dir=local_dir,\n", " allow_patterns=allow_patterns,\n", " )\n", " if _is_notebook():\n", " with _notebook_tqdm_area(\"DOWNLOAD PROGRESS\") as tqdm_class:\n", " result = _snapshot_download(**kwargs, tqdm_class=tqdm_class)\n", " else:\n", " result = _snapshot_download(**kwargs)\n", " if validator is not None:\n", " validator(local_dir)\n", " return result\n", "\n", " return _run_transfer(\"DOWNLOAD\", source_url, local_dir, action)\n", "\n", "\n", "def download_repo_fast(repo_id, local_dir, repo_type=\"model\"):\n", " return _download_snapshot(repo_id, local_dir, repo_type)\n", "\n", "\n", "def download_subfolder_fast(repo_id, remote_folder, local_dir, repo_type=\"model\"):\n", " remote_folder = _remote_path(remote_folder, \"REMOTE_FOLDER\")\n", "\n", " def validate(destination):\n", " expected = destination.joinpath(*_PurePosixPath(remote_folder).parts)\n", " if not expected.is_dir():\n", " raise FileNotFoundError(f\"Remote folder was not found or contained no downloadable files: {remote_folder}\")\n", "\n", " return _download_snapshot(\n", " repo_id,\n", " local_dir,\n", " repo_type,\n", " allow_patterns=f\"{remote_folder}/**\",\n", " validator=validate,\n", " )\n", "\n", "\n", "def download_files_fast(repo_id, remote_files, local_dir, repo_type=\"model\"):\n", " if isinstance(remote_files, (str, _Path)):\n", " remote_files = [remote_files]\n", " if not remote_files:\n", " raise ValueError(\"REMOTE_FILES must contain at least one file path.\")\n", " files = [_remote_path(item, \"REMOTE_FILES\") for item in remote_files]\n", " files = list(dict.fromkeys(files))\n", "\n", " def validate(destination):\n", " missing = [\n", " item\n", " for item in files\n", " if not destination.joinpath(*_PurePosixPath(item).parts).is_file()\n", " ]\n", " if missing:\n", " raise FileNotFoundError(\"Remote file(s) not found: \" + \", \".join(missing))\n", "\n", " return _download_snapshot(repo_id, local_dir, repo_type, allow_patterns=files, validator=validate)\n", "\n", "\n", "_PROGRESS_MODE = \"enhanced notebook display\" if _UPLOAD_DISPLAY_COMPATIBLE else \"upstream fallback display\"\n", "print(\n", " f\"TRANSFER ENGINE READY: huggingface_hub {_hub.__version__}, \"\n", " f\"hf_xet {_XET_VERSION_TEXT}, high-performance mode ON, {_PROGRESS_MODE}\"\n", ")\n" ] }, { "cell_type": "markdown", "id": "8d3b9d0e", "metadata": {}, "source": [ "## 2. Sign in\n", "\n", "Run once. Paste a Hugging Face token in the clearly marked value below, or leave it blank to reuse an existing login or start secure browser/device login. Uploads require a token with write access.\n", "\n", "**Never distribute a notebook after saving a real token inside it.**" ] }, { "cell_type": "code", "execution_count": null, "id": "ba34b6e1", "metadata": {}, "outputs": [], "source": [ "\n", "HF_TOKEN = \"\" # Enter for faster download, mandatory for upload, mandatory for private repos, if you don't enter it will use cached / active token if exists\n", "\n", "\n", "from huggingface_hub import login, whoami\n", "\n", "_token = HF_TOKEN.strip()\n", "login(token=_token or None, skip_if_logged_in=True)\n", "_identity = whoami()\n", "print(f\"AUTHENTICATED: {_identity['name']}\")\n", "del HF_TOKEN, _token, _identity" ] }, { "cell_type": "markdown", "id": "f978afe5", "metadata": {}, "source": [ "## 3. Recommended — very fast resumable folder upload\n", "\n", "Uses the current streamed, multi-commit `upload_folder` pipeline. Uploading begins while the folder is still being prepared; Xet handles chunking, deduplication, retries, and resume behavior. Progress shows separate rolling rates: **effective** is logical file data processed per second, while **network** is new data physically sent after deduplication. Wait for **UPLOAD COMPLETED**. Re-run the cell after an interruption to resume and skip data already committed or uploaded." ] }, { "cell_type": "code", "execution_count": null, "id": "6ae512f3", "metadata": {}, "outputs": [], "source": [ "# ========== EDIT ONLY THESE VALUES ==========\n", "REPO_ID = \"YourUserName/repo-name\"\n", "FOLDER_PATH = r\"/path/to/local/folder\"\n", "REPO_TYPE = \"model\" # model, dataset, or space\n", "# =============================================\n", "\n", "if \"upload_folder_fast\" not in globals():\n", " raise RuntimeError(\"SETUP REQUIRED: run Section 1 first.\")\n", "\n", "upload_folder_fast(REPO_ID, FOLDER_PATH, REPO_TYPE)" ] }, { "cell_type": "markdown", "id": "e5e3fb7b", "metadata": {}, "source": [ "## 4. Upload one file\n", "\n", "Set `REMOTE_FILE` to a repository path to choose a different remote name, or leave it blank to use the local file's basename automatically. Wait for **UPLOAD COMPLETED**. Re-running after an interruption lets Xet reuse uploaded chunks." ] }, { "cell_type": "code", "execution_count": null, "id": "6958b636", "metadata": {}, "outputs": [], "source": [ "# ========== EDIT ONLY THESE VALUES ==========\n", "REPO_ID = \"YourUserName/repo-name\"\n", "LOCAL_FILE = r\"/path/to/local/model.safetensors\"\n", "REMOTE_FILE = \"\" # blank = use LOCAL_FILE's basename; or set \"folder/name.safetensors\"\n", "REPO_TYPE = \"model\" # model, dataset, or space\n", "# =============================================\n", "\n", "if \"upload_file_fast\" not in globals():\n", " raise RuntimeError(\"SETUP REQUIRED: run Section 1 first.\")\n", "\n", "upload_file_fast(REPO_ID, LOCAL_FILE, REMOTE_FILE, REPO_TYPE)" ] }, { "cell_type": "markdown", "id": "fe5a6724", "metadata": {}, "source": [ "## 5. Download an entire repository\n", "\n", "Progress updates in place. Wait for **DOWNLOAD COMPLETED**. Re-running resumes from the local metadata cache and downloads only missing or changed data." ] }, { "cell_type": "code", "execution_count": null, "id": "2bf668a6", "metadata": {}, "outputs": [], "source": [ "# ========== EDIT ONLY THESE VALUES ==========\n", "REPO_ID = \"YourUserName/repo-name\"\n", "LOCAL_DIR = r\"/path/to/download/folder\"\n", "REPO_TYPE = \"model\" # model, dataset, or space\n", "# =============================================\n", "\n", "if \"download_repo_fast\" not in globals():\n", " raise RuntimeError(\"SETUP REQUIRED: run Section 1 first.\")\n", "\n", "download_repo_fast(REPO_ID, LOCAL_DIR, REPO_TYPE)" ] }, { "cell_type": "markdown", "id": "1e963d23", "metadata": {}, "source": [ "## 6. Download one remote subfolder\n", "\n", "Enter the folder path only; the helper adds the recursive match pattern. Do not add a leading slash or `/*`. Wait for **DOWNLOAD COMPLETED**. Re-run after an interruption to resume." ] }, { "cell_type": "code", "execution_count": null, "id": "c7c6e4eb", "metadata": {}, "outputs": [], "source": [ "# ========== EDIT ONLY THESE VALUES ==========\n", "REPO_ID = \"YourUserName/repo-name\"\n", "REMOTE_FOLDER = \"folder/path/in/repo\" # no leading slash and no /*\n", "LOCAL_DIR = r\"/path/to/download/folder\"\n", "REPO_TYPE = \"model\" # model, dataset, or space\n", "# =============================================\n", "\n", "if \"download_subfolder_fast\" not in globals():\n", " raise RuntimeError(\"SETUP REQUIRED: run Section 1 first.\")\n", "\n", "download_subfolder_fast(REPO_ID, REMOTE_FOLDER, LOCAL_DIR, REPO_TYPE)" ] }, { "cell_type": "markdown", "id": "a5933f80", "metadata": {}, "source": [ "## 7. Download one or more specific files\n", "\n", "File paths are relative to the repository root. Duplicate entries are removed automatically. Progress updates in place. Wait for **DOWNLOAD COMPLETED**. Re-run after an interruption to resume." ] }, { "cell_type": "code", "execution_count": null, "id": "b5ab776a", "metadata": {}, "outputs": [], "source": [ "# ========== EDIT ONLY THESE VALUES ==========\n", "REPO_ID = \"YourUserName/repo-name\"\n", "REMOTE_FILES = [\n", " \"model.safetensors\",\n", " \"subfolder/config.json\",\n", "]\n", "LOCAL_DIR = r\"/path/to/download/folder\"\n", "REPO_TYPE = \"model\" # model, dataset, or space\n", "# =============================================\n", "\n", "if \"download_files_fast\" not in globals():\n", " raise RuntimeError(\"SETUP REQUIRED: run Section 1 first.\")\n", "\n", "download_files_fast(REPO_ID, REMOTE_FILES, LOCAL_DIR, REPO_TYPE)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.11" } }, "nbformat": 4, "nbformat_minor": 5 }