mirrors
/
dedupe-linker
mirror of https://github.com/vbatts/dedupe-linker.git
1
0
Fork 0
Code Releases Activity
Utility for local deduplication of filesystems
35 Commits 1 Branch 0 Tags 55 KiB
Go 100%
Go to file
Vincent Batts 1db230aad4 file/hash: less noisy debug output 
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
 		2023-04-12 11:35:40 -04:00
base *.go: firm up whether it is the same file 2023-04-11 14:31:28 -04:00
cryptomap cryptomap: discoverable mapping and defaults 2016-09-13 11:10:24 -04:00
file file/hash: less noisy debug output 2023-04-12 11:35:40 -04:00
walker walker: golint 2016-09-13 11:10:17 -04:00
LICENSE LICENSE: add LICENSE 2016-09-08 10:38:32 -04:00
NOTES.md notes and base work 2014-09-17 09:21:56 -04:00
README.md README: update the usage and remove Travis 2023-04-12 11:25:50 -04:00
go.mod main: simple Usage description and add a go modules file 2023-04-11 14:14:54 -04:00
main.go *.go: ignoreSuffixes to not walk the basedir 2023-04-12 10:59:00 -04:00

README.md

dedupe-linker

Dedupe regular files on a filesystem by hard linking and a creating a content-addressible look-aside.

This defaults to sha1 checksums, but you can specify others like sha256 or sha512.

This also means that the comparison on whether something is a duplicate is based on checksum, and does not consider mode, owner, or xattrs.

This is really a pet-project, but changes welcome. And is only focused on Linux compatability, but changes welcome.

Install

go install github.com/vbatts/dedupe-linker@latest

Usage

The default look-aside base directory is in ~/.dedupe-linker, so the following would dedupe your home directory:

dedupe-linker -w $(nproc) ~/

Cleaning up from the look-aside needs only to check the links as a ref-count:

find ~/.dedupe-linker/ -type f -links 1

Then delete these files that only have 1 reference:

find ~/.dedupe-linker/ -type f -links 1 -exec rm -f "{}" \;