跳至主要內容

關於最強壓縮方式的調查

· 5 分鐘閱讀
ひかり

結論​

先說結論。

調查了對總計 34 GiB 的大量檔案進行壓縮與解壓時的時間與大小。

壓縮・建立封存檔​

方式指令realusersys壓縮後大小
tartar cf large-pkg.tar large-pkg1m19.449s0m6.702s0m48.121s26 GiB
tar.gztar czf large-pkg.tar.gz large-pkg11m33.942s11m18.811s0m55.573s5.5 GiB
LZ4tar cf - large-pkg | lz4 > large-pkg.tar.lz43m33.187s0m53.958s2m57.122s8.6 GiB
zstdtar cf - large-pkg | zstd -T0 -o large-pkg.tar.zst9m13.819s1m45.049s2m43.609s4.8 GiB
bzip2tar cf - large-pkg.1 | bzip2 > large-pkg.tar.bz237m22.743s28m40.329s3m31.000s4.3 GiB
xztar cf - large-pkg | xz > large-pkg.tar.xz125m31.447s124m10.523s5m18.330s3.3 GiB

展開 (解壓)​

方式指令realusersys
tartar xf large-pkg.tar2m11.793s0m6.906s2m4.183s
tar.gztar xf large-pkg.tar.gz3m39.544s2m1.189s2m58.317s
tar.gz（gzip）gzip -dc large-pkg.tar.gz | tar xf -3m40.416s2m0.272s3m0.043s
tar.gz（pigz）pigz -dc large-pkg.tar.gz | tar xf -3m53.711s1m38.147s4m42.893s
LZ4lz4 -dc large-pkg.tar.lz4 | tar xf -4m46.576s0m32.174s4m36.055s
zstdzstd -dc large-pkg.tar.zst | tar xf -3m46.419s0m46.533s3m34.668s
bzip2bzip2 -dc large-pkg.tar.bz2 | tar xf -11m31.287s9m52.644s4m17.974s
xzxz -dc large-pkg.tar.xz | tar xf -8m11.527s3m45.562s7m15.109s

準備大量小型檔案​

首先，作為大量小型檔案，使用 node_modules。

把 package.json 設成如下。套件隨意挑選。

package.json
{
  "name": "large-pkg",
  "version": "1.0.0",
  "description": "",
  "author": "",
  "type": "commonjs",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "dependencies": {
    "async": "^3.2.6",
    "axios": "^1.13.2",
    "bcryptjs": "^3.0.3",
    "bluebird": "^3.7.2",
    "body-parser": "^2.2.2",
    "chalk": "^5.6.2",
    "chalk-template": "^1.1.2",
    "cheerio": "^1.1.2",
    "chokidar": "^5.0.0",
    "commander": "^14.0.2",
    "cookie": "^1.1.1",
    "core-js": "^3.47.0",
    "cors": "^2.8.5",
    "debug": "^4.4.3",
    "dotenv": "^17.2.3",
    "express": "^5.2.1",
    "fast-glob": "^3.3.3",
    "form-data": "^4.0.5",
    "glob": "^13.0.0",
    "got": "^14.6.6",
    "inquirer": "^13.2.0",
    "jsonwebtoken": "^9.0.3",
    "lodash": "^4.17.21",
    "mime": "^4.1.0",
    "minimist": "^1.2.8",
    "mkdirp": "^3.0.1",
    "mkdirp-classic": "^0.5.3",
    "mongoose": "^9.1.4",
    "ms": "^2.1.3",
    "node-fetch": "^3.3.2",
    "ora": "^9.0.0",
    "passport": "^0.7.0",
    "prop-types": "^15.8.1",
    "qs": "^6.14.1",
    "react": "^19.2.3",
    "react-dom": "^19.2.3",
    "request": "^2.88.2",
    "rimraf": "^6.1.2",
    "semver": "^7.7.3",
    "sharp": "^0.34.5",
    "socket.io": "^4.8.3",
    "supports-color": "^10.2.2",
    "tslib": "^2.8.1",
    "uuid": "^13.0.0",
    "ws": "^8.19.0",
    "xml2js": "^0.6.2",
    "yargs": "^18.0.0"
  },
  "devDependencies": {
    "@babel/cli": "^7.28.6",
    "@babel/plugin-transform-runtime": "^7.28.5",
    "@babel/preset-env": "^7.28.6",
    "@babel/runtime": "^7.28.6",
    "autoprefixer": "^10.4.23",
    "ava": "^6.4.1",
    "babel-core": "^6.26.3",
    "babel-loader": "^10.0.0",
    "chai": "^6.2.2",
    "commitlint": "^20.3.1",
    "concurrently": "^9.2.1",
    "conventional-changelog": "^7.1.1",
    "cross-env": "^10.1.0",
    "css-loader": "^7.1.2",
    "dotenv-expand": "^12.0.3",
    "eslint": "^8.57.1",
    "eslint-config-prettier": "^10.1.8",
    "eslint-config-standard": "^17.1.0",
    "eslint-plugin-import": "^2.32.0",
    "eslint-plugin-jsx-a11y": "^6.10.2",
    "eslint-plugin-node": "^11.1.0",
    "eslint-plugin-prettier": "^5.5.5",
    "eslint-plugin-react": "^7.37.5",
    "husky": "^9.1.7",
    "jest": "^30.2.0",
    "less": "^4.5.1",
    "lint-staged": "^16.2.7",
    "mocha": "^11.7.5",
    "nodemon": "^3.1.11",
    "playwright": "^1.57.0",
    "pm2": "^6.0.14",
    "postcss": "^8.5.6",
    "prettier": "^3.8.0",
    "puppeteer": "^24.35.0",
    "rollup": "^4.55.1",
    "rxjs": "^7.8.2",
    "sass": "^1.97.2",
    "semantic-release": "^25.0.2",
    "sinon": "^21.0.1",
    "style-loader": "^4.0.0",
    "stylus": "^0.64.0",
    "supertest": "^7.2.2",
    "tailwindcss": "^4.1.18",
    "ts-loader": "^9.5.4",
    "ts-node": "^10.9.2",
    "typescript": "^5.9.3",
    "vite": "^7.3.1",
    "vitest": "^4.0.17",
    "webpack": "^5.104.1",
    "webpack-cli": "^6.0.1",
    "webpack-dev-server": "^5.2.3",
    "zx": "^8.8.5"
  }
}

接著用以下指令建立 node_modules。

npm i

來看看容量。

$ du -h -d1
566M    ./node_modules
567M    .

這樣就知道已建立大量小型檔案。

複製 node_modules 以建立更多檔案。

for i in {1..59}; do
    echo "node_modules.${i} をコピー中..."
    cp -r node_modules "node_modules.${i}"
done

再看一次容量。

$ du -h -d1
...
34G     .

變成相當大的容量。

tarball​

看看製作 tarball 的速度。

建立檔案​

$ time tar cf large-pkg.tar large-pkg

real    1m19.449s
user    0m6.702s
sys     0m48.121s

建立後 26 GiB

解壓​

$ time tar xf large-pkg.tar

real    2m11.793s
user    0m6.906s
sys     2m4.183s

打包與解壓都算快。

tar.gz​

接著用 tar.gz 測試。

tar 指令是用來打包的命令，gzip 是壓縮單一檔案的命令。把兩者結合所產生的就是 tar.gz 檔。現在 tar 指令本身就能壓縮與解壓 tar.gz（其他壓縮格式也同理）。

壓縮​

$ time tar czf large-pkg.tar.gz large-pkg

real    11m33.942s
user    11m18.811s
sys     0m55.573s

壓縮後：5.5 GiB

解壓​

$ time tar xf large-pkg.tar.gz

real    3m39.544s
user    2m1.189s
sys     2m58.317s

用 tar 與 gzip 解壓​

$ time sh -c 'gzip -dc large-pkg.tar.gz | tar xf -'

real    3m40.416s
user    2m0.272s
sys     3m0.043s

速度幾乎沒差。

並行解壓 (pigz)​

$ time sh -c 'pigz -dc large-pkg.tar.gz | tar xf -'

real    3m53.711s
user    1m38.147s
sys     4m42.893s

速度沒有太大改變。看起來檔案讀寫速度成為瓶頸。

bzip2​

壓縮​

$ time sh -c 'tar cf - large-pkg.1 | bzip2 > large-pkg.tar.bz2'

real    37m22.743s
user    28m40.329s
sys     3m31.000s

壓縮後：4.3 GiB

壓縮花了比較多時間，但壓縮率還不錯。

解壓​

$ time sh -c 'bzip2 -dc large-pkg.tar.bz2 | tar xf -'

real    11m31.287s
user    9m52.644s
sys     4m17.974s

解壓也花不少時間，但解壓率還不錯。

xz​

壓縮​

$ time sh -c 'tar cf - large-pkg | xz > large-pkg.tar.xz'

real    125m31.447s
user    124m10.523s
sys     5m18.330s

壓縮後：3.3 GiB

壓縮率極為優秀，但花的時間太多。

如果網路極度慢、儲存成本高、且多年才用一次的罕見情境，或許可以考慮。

解壓​

$ time sh -c 'xz -dc large-pkg.tar.xz | tar xf -'

real    8m11.527s
user    3m45.562s
sys     7m15.109s

LZ4​

壓縮​

$ time sh -c 'tar cf - large-pkg | lz4 > large-pkg.tar.lz4'

real    3m33.187s
user    0m53.958s
sys     2m57.122s

解壓​

$ time sh -c 'lz4 -dc large-pkg.tar.lz4 | tar xf -'

real    4m46.576s
user    0m32.174s
sys     4m36.055s

zstd​

Meta（前 Facebook）開發的高速壓縮方式。

壓縮​

$ time sh -c 'tar cf - large-pkg | zstd -T0 -o large-pkg.tar.zst'
/*stdin*\            : 18.57%   (27492075520 => 5106239218 bytes, large-pkg.tar.zst)

real    9m13.819s
user    1m45.049s
sys     2m43.609s

解壓​

$ time sh -c 'zstd -dc large-pkg.tar.zst | tar xf -'
large-pkg.tar.zst   : 27492075520 bytes

real    3m46.419s
user    0m46.533s
sys     3m34.668s

コメント

読み込み中...

コメントを投稿する

★★★★★