以下来自:
Git LFS
…that reduces the impact of large files in your repository by downloading the relevant versions of them
lazily
. Specifically,
large files are downloaded during the checkout process
rather than during cloning or fetching.
Git LFS does this by
replacing large files in your repository with tiny
pointer
files
. During normal usage, you’ll never see these pointer files as they are handled automatically by Git LFS:
-
When you add a file to your repository, Git LFS replaces its contents with a pointer, and stores the file contents in a
local Git LFS cache
.
-
When you push new commits to the server, any Git LFS files referenced by the newly pushed commits are transferred from your local Git LFS cache to the
remote Git LFS store
tied to your Git repository.
-
When you
checkout a commit
that contains Git LFS pointers, they are replaced with files from your local Git LFS cache, or downloaded from the remote Git LFS store.
To use Git LFS, you will need
a Git LFS aware host
such as
Bitbucket Cloud
or
Bitbucket Server
. Repository users will need to have the
Git LFS command-line client installed
, or a Git LFS aware GUI client such as
Sourcetree
.
To create a new Git LFS aware repository, you’ll need to run
git lfs install
after you create the repository:
$ mkdir Atlasteroids
$ cd Atlasteroids
$ git init
Initialized empty Git repository in /Users/tpettersen/Atlasteroids/.git/
$ git lfs install
Updated pre-push hook.
Git LFS initialized.
This installs a special pre-push
Git hook in your repository that will transfer Git LFS files to the server when you git push
.
Once Git LFS is initialized for your repository, you can specify which files to track using git lfs track
.
Once Git LFS is installed, you can clone a Git LFS repository as normal using git clone
. At the end of the cloning process Git will check out the default branch (usually master
), and any Git LFS files needed to complete the checkout process will be automatically downloaded for you. For example:
$ git clone git@bitbucket.org:tpettersen/Atlasteroids.git
When running git clone, Git LFS files are downloaded one at a time as pointer files are checked out of your repository.
If you’re cloning a repository with a large number of LFS files, the explicit git lfs clone
command offers far better performance:
$ git lfs clone git@bitbucket.org:tpettersen/Atlasteroids.git
Rather than downloading Git LFS files one at a time, the git lfs clone
command waits until the checkout is complete, and then downloads any required Git LFS files as a batch. This takes advantage of parallelized downloads, and dramatically reduces the number of HTTP requests and processes spawned (which is especially important for improving performance on Windows).
Just like cloning, you can pull from a Git LFS repository using a normal git pull
. Any needed Git LFS files will be downloaded as part of the automatic checkout process once the pull completes:
$ git pull
No explicit commands are needed to retrieve Git LFS content. However, if the checkout fails for an unexpected reason, you can download any missing Git LFS content for the current commit with git lfs pull
:
$ git lfs pull
ike git lfs clone
, git lfs pull
downloads your Git LFS files as a batch. If you know a large number of files have changed since the last time you pulled, you may wish to disable the automatic Git LFS download during checkout, and then batch download your Git LFS content with an explicit git lfs pull
. This can be done by overriding your Git config with the -c
option when you invoke git pull
:
$ git -c filter.lfs.smudge= -c filter.lfs.required=false pull && git lfs pull
Since that’s rather a lot of typing, you may wish to create a simple Git alias to perform a batched Git and Git LFS pull for you:
$ git config --global alias.plfs "\!git -c filter.lfs.smudge= -c filter.lfs.required=false pull && git lfs pull"
$ git plfs
When you add a new type of large file to your repository, you’ll need to tell Git LFS to track it by specifying a pattern using the git lfs track
command:
$ git lfs track "*.ogg"
Tracking *.ogg
Note that the quotes around "*.ogg"
are important. Omitting them will cause the wildcard to be expanded by your shell, and individual entries will be created for each .ogg
file in your current directory:
The patterns supported by Git LFS are the same as those supported by .gitignore
, for example:
$ git lfs track "*.ogg"
$ git lfs track "music.ogg"
$ git lfs track "Assets/"
$ git lfs track "Assets/*"
$ git lfs track "Assets/Audio/*.ogg"
$ git lfs track "**/Music/*.ogg"
$ git lfs track "*xxhdpi*.png
These patterns are relative to the directory in which you ran the git lfs track
command. To keep things simple, it is best to run git lfs track
from the root of your repository. Note that Git LFS does not support negative patterns like .gitignore
does.
After running git lfs track
, you’ll notice a new file named .gitattributes
in the directory you ran the command from. .gitattributes
is a Git mechanism for binding special behaviors to certain file patterns. Git LFS automatically creates or updates .gitattributes
files to bind tracked file patterns to the Git LFS filter. However, you will need to commit any changes to the .gitattributes
file to your repository yourself:
$ git lfs track "*.ogg"
Tracking *.ogg
$ git add .gitattributes
$ git diff --cached
diff --git a/.gitattributes b/.gitattributes
new file mode 100644
index 0000000..b6dd0bb
--- /dev/null
+++ b/.gitattributes
@@ -0,0 +1 @@
+*.ogg filter=lfs diff=lfs merge=lfs -text
$ git commit -m "Track ogg files with Git LFS"
For ease of maintenance, it is simplest to keep all Git LFS patterns in a single .gitattributes
file by always running git lfs track
from the root of your repository. However, you can display a list of all patterns that are currently tracked by Git LFS (and the .gitattributes
files they are defined in) by invoking git lfs track
with no arguments:
$ git lfs track
You can stop tracking a particular pattern with Git LFS by simply removing the appropriate line from your .gitattributes
file, or by running the git lfs untrack
command:
$ git lfs untrack "*.ogg"
You can commit and push as normal to a repository that contains Git LFS content. If you have committed changes to files tracked by Git LFS, you will see some additional output from git push
as the Git LFS content is transferred to the server:
$ git push
If transferring the LFS files fails for some reason, the push will be aborted and you can safely try again. Like Git, Git LFS storage is content addressable: content is stored against a key which is a SHA-256 hash of the content itself. This means it is always safe to re-attempt transferring Git LFS files to the server; you can’t accidentally overwrite a Git LFS file’s contents with the wrong version.
To migrate a Git LFS repository from one hosting provider to another, you can use a combination of git lfs fetch
and git lfs push
with the --all option
specified.
$ git clone --bare git@github.com:kannonboy/atlasteroids.git
$ cd atlasteroids
$ git remote add bitbucket git@bitbucket.org:tpettersen/atlasteroids.git
$ git remote add github git@github.com:kannonboy/atlasteroids.git
$ git lfs fetch --all github
$ git push --mirror bitbucket
$ git lfs push --all bitbucket
Moving a Git LFS repository between hosts, you can also elect to fetch all Git LFS content for your repository with git lfs fetch --all
You can delete files from your local Git LFS cache with the git lfs prune
command:
$ git lfs prune
This will delete any local Git LFS files that are considered old. An old file is any file not referenced by:
- the currently checked out commit
- a commit that has not yet been pushed (to origin, or whatever
lfs.pruneremotetocheck
is set to) - a recent commit
By default, a recent commit is any commit created in the last tendays. This is calculated by adding:
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-aXgdCVGD-1578042723742)(picture\08.svg)]
You can configure the prune offset to retain Git LFS content for a longer period:
$ git config lfs.pruneoffsetdays 21
Unlike Git’s built-in garbage collection, Git LFS content is not pruned automatically, so running git lfs prune
on a regular basis is a good idea to keep your local repository size down.
You can test out what effect a prune operation will have with git lfs prune --dry-run
:
$ git lfs prune --dry-run
✔ 4 local objects, 33 retained
4 files would be pruned (2.1 MB)
And exactly which Git LFS objects will be pruned with git lfs prune --verbose --dry-run
:
$ git lfs prune --dry-run --verbose
✔ 4 local objects, 33 retained
4 files would be pruned (2.1 MB)
* 4a3a36141cdcbe2a17f7bcf1a161d3394cf435ac386d1bff70bd4dad6cd96c48 (2.0 MB)
* 67ad640e562b99219111ed8941cb56a275ef8d43e67a3dac0027b4acd5de4a3e (6.3 KB)
* 6f506528dbf04a97e84d90cc45840f4a8100389f570b67ac206ba802c5cb798f (1.7 MB)
* a1d7f7cdd6dba7307b2bac2bcfa0973244688361a48d2cebe3f3bc30babcf1ab (615.7 KB)
The long hexadecimal strings output by --verbose
mode are SHA-256 hashes (also known as Object IDs, or OIDs) of the Git LFS objects to be pruned. You can use the techniques described in Finding paths or commits that reference a Git LFS object to find our more about the objects that will be pruned.
As an additional safety check, you can use the --verify-remote
option to check whether the remote Git LFS store has a copy of your Git LFS objects before they are pruned:
$ git lfs prune --verify-remote
✔ 16 local objects, 2 retained, 12 verified with remote
Pruning 14 files, (1.7 MB)
✔ Deleted 14 files
This makes the pruning process significantly slower, but gives you peace of mind knowing that any pruned objects are recoverable from the server. You can enable the --verify-remote
option permanently for your system by configuring the lfs.pruneverifyremotealways
property globally:
$ git config --global lfs.pruneverifyremotealways true
Or you can enable remote verification for just the context repository by omitting the --global
option from the command above.
The Git LFS command-line client doesn’t support pruning files from the server, so how you delete them depends on your hosting provider.
If you have a Git LFS SHA-256 OID, you can determine which commits reference it with git log --all -p -S <OID>
:
$ git log --all -p -S 3b6124b8b01d601fa20b47f5be14e1be3ea7759838c1aac8f36df4859164e4cc
This git log
incantation generates a patch (-p
) from commits on any branch (--all
) that add or remove a line (-S
) containing the specified string (a Git LFS SHA-256 OID).
The patch shows you the commit and the path to the LFS object, as well as who added it, and when it was committed. You can simply checkout the commit, and Git LFS will download the file if needed and place it in your working copy.
If you suspect that a particular Git LFS object is in your current HEAD, or on a particular branch, you can use git grep
to find the file path that references it:
$ git grep 3b6124b8b01d601fa20b47f5be14e1be3ea7759838c1aac8f36df4859164e4cc HEAD
HEAD:Assets/Sprites/projectiles-spritesheet.png:oid sha256:3b6124b8b01d601fa20b47f5be14e1be3ea7759838c1aac8f36df4859164e4cc
$ git grep e88868213a5dc8533fc9031f558f2c0dc34d6936f380ff4ed12c2685040098d4 power-ups
power-ups:Assets/Sprites/shield2.png:oid sha256:e88868213a5dc8533fc9031f558f2c0dc34d6936f380ff4ed12c2685040098d4
In some situations you may want to only download a subset of the available Git LFS content for a particular commit.
You can exclude a pattern or subdirectory using git lfs fetch -X
(or --exclude
):
$ git lfs fetch -X "Assets/**"
Alternatively, you may want to only include a particular pattern or subdirectory. For example, an audio engineer could fetch just ogg
and wav
files with git lfs fetch -I
(or --include
):
$ git lfs fetch -I "*.ogg,*.wav"
If you combine includes and excludes, only files that match an include pattern and do not match an exclude pattern will be fetched.
$ git lfs fetch -I "Assets/**" -X "*.gif"
Excludes and includes support the same patterns as git lfs track
and .gitignore
. You can make these patterns permanent for a particular repository by setting the lfs.fetchinclude
and lfs.fetchexclude
config properties:
$ git config lfs.fetchinclude "Assets/**"
$ git config lfs.fetchexclude "*.gif"
These settings can also be applied to every repository on your system by appending the --global
option.
Unfortunately, there is no easy way of resolving binary merge conflicts. With Git LFS file locking, you can lock files by extension or by file name and prevent binary files from being overwritten during a merge.
In order to take advantage of LFS’ file locking feature, you first need to tell Git which type of files are lockable. In the example below, the --lockable
flag is appended to the git lfs track
command which both stores PSD files in LFS and marks them as lockable.
$ git lfs track "*.psd" --lockable
Then add the following to your .gitattributes file:
*.psd filter=lfs diff=lfs merge=lfs -text lockable
When preparing to make changes to an LFS file, you’ll use the lock command in order to register the file as locked on your Git server.
$ git lfs lock images/foo.psd
Locked images/foo.psd
Once you no longer need the file lock, you can remove it using Git LFS’ unlock command.
$ git lfs unlock images/foo.psd
Git LFS file locks can be overridden, similar to git push
, using a --force
flag. Do not use the --force
flag unless you’re absolutely sure you know what you’re doing.
$ git lfs unlock images/foo.psd --force
以下内容来自:Git LFS 操作指南
git lfs track
git lfs track "*.psd"
git lfs untrack "*.psd"
git lfs status
git lfs ls-files
git lfs version
git lfs clone https://github.com/user/repo.git
[1] git lfs clone
通过合并获取 LFS 对象的请求,减少了 LFS API 的调用,并行化 LFS 对象的下载,从而达到显著的速度提升。git lfs clone
命令同样也兼容没有使用 LFS 的仓库。即无论要克隆的仓库是否使用 LFS,都可以使用 git lfs clone
命令来进行克隆。
[2] 目前最新版本的 git clone
已经能够提供与 git lfs clone
一致的性能,因此自 Git LFS 2.3.0 版本起,git lfs clone
已不再推荐使用。
使用 Git LFS 的核心思想就是把需要进行版本管理、但又占用很大空间的那部分文件独立于 Git 仓库进行管理。从而加快克隆仓库本身的速度,同时获得灵活的管理 LFS 对象的能力。
默认情况下,只有当前 commit 下的 LFS 对象的当前版本才会被获取。
如果自己的相关工作不涉及到被 Git LFS 所管理的文件的话,可以选择只获取 Git 仓库自身的内容,而完全跳过 LFS 对象的获取。
GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/user/repo.git
git -c filter.lfs.smudge= -c filter.lfs.required=false clone https://github.com/user/repo.git
注:GIT_LFS_SKIP_SMUDGE=1
及 git -c filter.lfs.smudge= -c filter.lfs.required=false
同样使用于其他 git 命令,如 checkout
, reset
等。
-
Git LFS 对象在本地仓库的存放位置?
通过 Git LFS 所管理的对象实际在本地的存储位置是在 .git/lfs/objects
目录下,该目录根据对象的 sha256 值来组织。
作为对比,Git 自身所管理的对象则是存储在 .git/objects
目录下,根据 commit, tree, blob, tag 的 sha1 值来组织。
-
已经使用 git lfs track somefile
追踪了某个文件,但该文件并未以 LFS 存储。
如果被 LFS 追踪管理的文件的大小为 0 的话,则该文件不会以 LFS 的形式存储起来。
只有当一个文件至少有 1 个字节时,其才会以 LFS 的形式存储。
注:一般使用 LFS 时,我们也不会用其追踪空文件,即使追踪了空文件,对于使用也没有任何影响。提到这点主要是为了消除在测试使用 LFS 时可能遇到的困惑。
notes:
-
注意顺序 git lfs track xxx
-> git add .gitattributes
-> git commit
-> git add xxx
-> git commit
-> git push -u origin master
-
git lfs ls-files
在 git lfs track xxx
且 git add xxx
后可以看到当前追踪的所有文件
-
GitHub文件限制:
关于存储和带宽使用情况
每个使用 Git Large File Storage 的帐户可获得 1 GB 的免费存储空间和 1 GB 的一个月免费带宽。如果带宽和存储配额不足,您可以选择为 Git LFS 购买附加配额。
Git LFS 是适用于 GitHub 上每个仓库的变量,无论您的帐户或组织是否有付费的订阅。
以下来自:Git LFS 服务器实现杂谈
在 git 存储库中,有一些特殊文件, .gitattributes
,.gitignore
,.gitattributes
顾名思义用来管理存储库下路径的属性,如何过滤,如何 diff,如何合并。而 .gitignore
的作用则是排除指定的路径,不添加到版本库中。
笔者最早了解到 .gitattributes 时是使用 git 管理毕业论文,设置好 .gitattributes 后,就可以到 docx 文档进行 diff 了。利用这一特性,git lfs 在运行命令 git lfs track
后,将路径添加到 .gitattributes
文件中,比如这样:
clang.tar.gz filter=lfs diff=lfs merge=lfs -text
然后修改 git 钩子,包括 post-checkout post-commit post-merge pre-push,其中的内容都是将当前钩子命令当做参数启动 git-lfs。
创建提交的过程中,比如添加了 clang.tar.gz ,git 会启动 lfs 命令,将 clang.tar.gz 文件的 hash 值(目前是 sha256),大小写入到一个名字叫 clang.tar.gz
文件中,然后将此文件作为大文件的 Pointer
提交到版本库中,原始的 clang.tar.gz 会单独存储到 lfs/objects
,将更新推送到远程服务器上时,pre-push 钩子启动 git lfs,git lfs 连接到远程的 LFS 服务器,将文件上传到服务器上。
而 checkout 或者 pull 的时候,本地没有的大文件,git lfs 就会将大文件下载下来。
简单的说就是利用 .gitattributes 偷梁换柱。
git lfs 规范可以在 https://github.com/git-lfs/git-lfs/blob/master/docs/spec.md 查看。
-
post-checkout:
#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/post-checkout.\n"; exit 2; }
git lfs post-checkout "$@"
-
post-commit:
#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/post-commit.\n"; exit 2; }
git lfs post-commit "$@"
-
post-merge:
#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/post-merge.\n"; exit 2; }
git lfs post-merge "$@"
-
pre-push:
#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/pre-push.\n"; exit 2; }
git lfs pre-push "$@"
以下来自:Git LFS Specification
The core Git LFS idea is that instead of writing large blobs to a Git repository, only a pointer file is written.
- Each line MUST be of the format
{key} {value}\n
(trailing unix newline). - The first key is always
version
. - Lines of key/value pairs MUST be sorted alphabetically in ascending order (with the exception of
version
, which is always first). - Pointer files MUST be stored in Git with their executable bit matching that of the replaced file.
An empty file is the pointer for an empty file. That is, empty files are passed through LFS without any change.
The required keys are:
version
is a URL that identifies the pointer file spec. Parsers MUST use simple string comparison on the version, without any URL parsing or normalization. It is case sensitive, and %-encoding is discouraged.oid
tracks the unique object id for the file, prefixed by its hashing method: {hash-method}:{hash}
. Currently, only sha256
is supported.size
is in bytes.
version https://hawser.github.com/spec/v1
oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
size 12345
(ending \n)
For testing compliance of any tool generating its own pointer files, the reference is this official Git LFS tool:
NOTE: exact pointer command behavior TBD!
-
Tools that parse and regenerate pointer files MUST preserve keys that they don’t know or care about.
-
Run the pointer
command to generate a pointer file for the given local file:
$ git lfs pointer --file=path/to/file
Git LFS pointer for path/to/file:
version https://git-lfs.github.com/spec/v1
oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
size 12345
-
Run pointer
to compare the blob OID of a pointer file built by Git LFS with a pointer built by another tool.
$ git lfs pointer --file=path/to/file --pointer=other/pointer/file
$ cat other/pointer/file | git lfs pointer --file=path/to/file --stdin
Git LFS uses the clean
and smudge
filters to decide which files use it. The global filters can be set up with git lfs install
These filters ensure that large files aren’t written into the repository proper, instead being stored locally at .git/lfs/objects/{OID-PATH}
(where {OID-PATH}
is a sharded filepath of the form OID[0:2]/OID[2:4]/OID
), synchronized with the Git LFS server as necessary.
The clean
filter runs as files are added to repositories. Git sends the content of the file being added as STDIN, and expects the content to write to Git as STDOUT.
- Stream binary content from STDIN to a temp file, while calculating its SHA-256 signature.
- Atomically move the temp file to
.git/lfs/objects/{OID-PATH}
if it does not exist, and the sha-256 signature of the contents matches the given OID. - Delete the temp file.
- Write the pointer file to STDOUT.
Note that the clean
filter does not push the file to the server. Use the git push
command to do that (lfs files are pushed before commits in a pre-push hook).
The smudge
filter runs as files are being checked out from the Git repository to the working directory. Git sends the content of the Git blob as STDIN, and expects the content to write to the working directory as STDOUT.
- Read 100 bytes.
- If the content is ASCII and matches the pointer file format:
- Look for the file in
.git/lfs/objects/{OID-PATH}
. - If it’s not there, download it from the server.
- Write its contents to STDOUT
- Otherwise, simply pass the STDIN out through STDOUT.
Deletes local copies of LFS files which are old, thus freeing up disk space.
Note: you should not run git lfs prune
if you have different repositories sharing the same custom storage directory; see git-lfs-config(1) for more details about lfs.storage
option.
-
lfs.storage
可以更改lfs文件的存储位置(不同仓库可以共享同一存储目录)
Allow override LFS storage directory. Non-absolute path is relativized to inside of Git repository directory (usually .git
).
Note: you should not run git lfs prune
if you have different repositories sharing the same storage directory.
Default: lfs
in Git repository directory (usually .git/lfs
).
drone克隆含子项目的仓库时:
kind: pipeline
type: docker
name: default
clone:
disable: true
steps:
- name: clone
image: alpine/git
commands:
- git lfs clone --recursive http://192.168.xxx.xxx:11080/username/repo.git
** 会报错,因为 alpine/git
镜像中没有安装 git-lfs
** 如果使用 git clone
命令,那么使用了 lfs
的子模块中的大文件不会被拉取下来
构建安装好 git-lfs
的镜像:
FROM alpine
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
RUN apk update && apk upgrade && \
apk add --no-cache bash openssh git git-lfs
** 更换源
-
文件过大,commit时自动打包
$ git commit -m "added lfw_160"
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
Enumerating objects: 7299, done.
Counting objects: 100% (7299/7299), done.
Delta compression using up to 4 threads
Compressing objects: 100% (7013/7013), done.
Writing objects: 100% (7299/7299), done.
Total 7299 (delta 0), reused 0 (delta 0)
Removing duplicate objects: 100% (256/256), done.
[master (root-commit) ef03f8f] added lfw_160
6029 files changed, 24112 insertions(+)
create mode 100644 .gitattributes
create mode 100644 lfw_160/Aaron_Sorkin/Aaron_Sorkin_0001.png
create mode 100644 lfw_160/Aaron_Sorkin/Aaron_Sorkin_0002.png
......
-
仓库克隆成功但检出失败
其中pre-trained-models
仓库push前没有打包。
进入仓库文件夹后,使用git checkout -f HEAD
有时会成功,而且有时会要求输入用户名和密码(使用http克隆时)
/
git version 2.22.0
/
Cloning into 'pre-trained-models'...
remote: Enumerating objects: 8, done.
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 8 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (8/8), done.
/
Cloning into 'lfw-dataset'...
remote: Enumerating objects: 7299, done.
remote: Counting objects: 100% (7299/7299), done.
remote: Compressing objects: 100% (7013/7013), done.
remote: Total 7299 (delta 0), reused 7299 (delta 0)
Receiving objects: 100% (7299/7299), 1006.13 KiB | 23.96 MiB/s, done.
Downloading lfw_160/Jorge_Moreno/Jorge_Moreno_0001.png (33 KB)
Error downloading object: lfw_160/Jorge_Moreno/Jorge_Moreno_0001.png (b8e0536): Smudge error: Error downloading lfw_160/Jorge_Moreno/Jorge_Moreno_0001.png (b8e05360ab5e1d92a8d2bf7f3eb2793ce6ddb21f1430fac9fb27663ad448b8cb): batch response: Repository or object not found: http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it
Errors logged to /lfw-dataset/.git/lfs/logs/20191128T110441.134505027.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: lfw_160/Jorge_Moreno/Jorge_Moreno_0001.png: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'
-
使用GIT_TRACE=1
查看克隆细节,服务器内部错误500
- 在Windows、Ubuntu平台(非docker容器中),有或没有像仓库添加ssh密钥的情况下,都克隆并检出成功
- 在docker容器中,Ubuntu、Alpine都克隆成功但检出失败,因此推断是由于docker容器的某种限制导
- 请求batch,服务器内部错误500 api error: Fatal error: Server error: http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
- 之后的数次请求,找不到对象404 api error: Repository or object not found: http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
- 重试几次后,最终返回 api error: Repository or object not found: http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it
09:46:03.806661 trace git-lfs: HTTP: 200
09:46:03.806706 trace git-lfs: HTTP: 200
09:46:03.807167 trace git-lfs: filepathfilter: accepting "lfw_160/Joschka_Fischer/Joschka_Fischer_0004.png"
09:46:03.807853 trace git-lfs: filepathfilter: accepting "lfw_160/Joseph_Deiss/Joseph_Deiss_0003.png"
09:46:03.807990 trace git-lfs: tq: sending batch of size 100
09:46:03.808246 trace git-lfs: api: batch 100 files
09:46:03.808293 trace git-lfs: HTTP: POST http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
09:46:03.808350 trace git-lfs: filepathfilter: accsepting "lfw_160/Jose_Serra/Jose_Serra_0003.png"s
09:46:03.808565 trace git-lfs: filepathfilter: accepting "lfw_160/Judi_Patton/Judi_Patton_0001.png"
09:46:04.138014 trace git-lfs: HTTP: 500
09:46:04.138057 trace git-lfs: api error: Fatal error: Server error: http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
09:46:04.138366 trace git-lfs: tq: sending batch of size 100
09:46:04.138465 trace git-lfs: api: batch 100 files
09:46:04.138511 trace git-lfs: HTTP: POST http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
09:46:04.147331 trace git-lfs: HTTP: 404
09:46:04.147371 trace git-lfs: api error: Repository or object not found: http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it
09:46:04.147630 trace git-lfs: tq: sending batch of size 100
09:46:04.147725 trace git-lfs: api: batch 100 files
09:46:04.147768 trace git-lfs: HTTP: POST http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
09:46:04.156188 trace git-lfs: HTTP: 404
09:46:04.156219 trace git-lfs: api error: Repository or object not found: http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it
09:46:04.361248 trace git-lfs: tq: sending batch of size 25
09:46:04.361318 trace git-lfs: api: batch 25 files
09:46:04.361373 trace git-lfs: HTTP: POST http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
09:46:04.370427 trace git-lfs: HTTP: 404
09:46:04.370465 trace git-lfs: api error: Repository or object not found: http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it
09:46:04.384150 trace git-lfs: filepathfilter: accepting "lfw_160/Julia_Tymoshenko/Julia_Tymoshenko_0003.png"
Downloading lfw_160/Julia_Tymoshenko/Julia_Tymoshenko_0003.png (32 KB)
09:46:04.384189 trace git-lfs: tq: running as batched queue, batch size of 100
09:46:04.384210 trace git-lfs: tq: sending batch of size 1
09:46:04.384255 trace git-lfs: api: batch 1 files
09:46:04.384301 trace git-lfs: HTTP: POST http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
09:46:04.393577 trace git-lfs: HTTP: 404
09:46:04.393609 trace git-lfs: api error: Repository or object not found: http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it
Error downloading object: lfw_160/Julia_Tymoshenko/Julia_Tymoshenko_0003.png (ab93cd4): Smudge error: Error downloading lfw_160/Julia_Tymoshenko/Julia_Tymoshenko_0003.png (ab93cd45d9ada70acdaa7e7d0d84b4155fa73cc36d1c070d1817d6631932fb5a): batch response: Repository or object not found: http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it
Errors logged to /lfw-dataset/.git/lfs/logs/20191130T094604.393673476.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: lfw_160/Julia_Tymoshenko/Julia_Tymoshenko_0003.png: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'
-
更详细的信息
GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1
GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git clone http://192.168.xxx.xxx:10080/usr/lfw-dataset.git
< HTTP/1.1 200 OK
< Content-Length: 28128
< Content-Type: application/octet-stream
< Date: Sun, 01 Dec 2019 02:20:39 GMT
< Set-Cookie: lang=en-US; Path=/; Max-Age=2147483647
< Set-Cookie: i_like_gitea=2c0a816af789763f; Path=/; HttpOnly
< Set-Cookie: _csrf=UXrpPl4bM6x53OXiXSd2EBdtkEc6MTU3NTE2NjgzOTgwMzQzNTk5Mw; Path=/; Expires=Mon, 02 Dec 2019 02:20:39 GMT; HttpOnly
< X-Frame-Options: SAMEORIGIN
02:20:39.816151 trace git-lfs: xfer: adapter "basic" worker 3 finished job for "6fbb3cfda70f7f4c147134a1fd159a124a07addde5cd97b3d9e602c8cc05099a"
02:20:39.816297 trace git-lfs: filepathfilter: accepting "lfw_160/Joschka_Fischer/Joschka_Fischer_0004.png"
02:20:39.817394 trace git-lfs: tq: sending batch of size 100
02:20:39.817485 trace git-lfs: api: batch 100 files
02:20:39.817520 trace git-lfs: HTTP: POST http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
> POST /usr/lfw-dataset.git/info/lfs/objects/batch HTTP/1.1
> Host: 192.168.xxx.xxx:10080
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 8871
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.7.2 (GitHub; linux amd64; go 1.12.5; git f3af9f768e)
{"operation":"download","objects":[{"oid":"fe755d8d795033b8fe4092c0e6baa0d908436069dd4503a1005315c3a7ad959f","size":46076},{"oid":"36c2029b8103ed6e5f8aa1fe94a047cb051ca2e26294c5b980dd91e10fbb84bf","size":43510},……{"oid":"3f7a2397513cc87d38fbf2f31f3be29f8edb9951f0d08a11cc386560d4263c2a","size":28813}],"ref":{"name":"refs/heads/master"}}02:20:40.198751 trace git-lfs: HTTP: 500
< HTTP/1.1 500 Internal Server Error
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8
< Date: Sun, 01 Dec 2019 02:20:40 GMT
< Set-Cookie: lang=en-US; Path=/; Max-Age=2147483647
< Set-Cookie: i_like_gitea=11ed8dfc2d08ef19; Path=/; HttpOnly
< Set-Cookie: _csrf=u0QaX5rU9M3YXlvQjiyA0B6RdX06MTU3NTE2NjgzOTgxNzc2OTY2NQ; Path=/; Expires=Mon, 02 Dec 2019 02:20:39 GMT; HttpOnly
< X-Frame-Options: SAMEORIGIN
02:20:40.198867 trace git-lfs: api error: Fatal error: Server error: http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
02:20:40.199269 trace git-lfs: tq: sending batch of size 100
02:20:40.199389 trace git-lfs: api: batch 100 files
02:20:40.199442 trace git-lfs: HTTP: POST http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
> POST /usr/lfw-dataset.git/info/lfs/objects/batch HTTP/1.1
> Host: 192.168.xxx.xxx:10080
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 8871
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.7.2 (GitHub; linux amd64; go 1.12.5; git f3af9f768e)
{"operation":"download","objects":[{"oid":"a892c6a142d8b246b27b6531b70dbba6005fa837014b2f1d95d95dc68d64b853","size":43791},{"oid":"21bc4e6cdc605e3b244767d138057537e3f136461bc915b9775e51849e35433c","size":42884},……{"oid":"2e66599d82999056aa8f02b9562369f971b3c4462c971953c9773dd8a1280ecd","size":28470}],"ref":{"name":"refs/heads/master"}}02:20:40.210304 trace git-lfs: HTTP: 404
< HTTP/1.1 404 Not Found
< Content-Length: 23
< Content-Type: text/plain; charset=utf-8
< Date: Sun, 01 Dec 2019 02:20:40 GMT
< Set-Cookie: lang=en-US; Path=/; Max-Age=2147483647
< Set-Cookie: i_like_gitea=09c73f17462dc4ec; Path=/; HttpOnly
< Set-Cookie: _csrf=fadTW0WNPRIHuwV3BtmAwAx6tWo6MTU3NTE2Njg0MDE5OTc4MjY1Mw; Path=/; Expires=Mon, 02 Dec 2019 02:20:40 GMT; HttpOnly
< X-Frame-Options: SAMEORIGIN
02:20:40.210406 trace git-lfs: api error: Repository or object not found: http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it
02:20:40.210752 trace git-lfs: tq: sending batch of size 100
02:20:40.210875 trace git-lfs: api: batch 100 files
02:20:40.210933 trace git-lfs: HTTP: POST http://192.168.xxx.xxx:10080/usr/lfw-dataset.git/info/lfs/objects/batch
> POST /usr/lfw-dataset.git/info/lfs/objects/batch HTTP/1.1
> Host: 192.168.xxx.xxx:10080
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 8871
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.7.2 (GitHub; linux amd64; go 1.12.5; git f3af9f768e)
{"operation":"download","objects":[{"oid":"f5a23061bf1b96a1d1f402edd9965cc54051be26f87e855d1089bbc3bcc8c6aa","size":42833},{"oid":"a7d9f41c34b042b535c6eeddd4a3df02a1829d59b31becc7b58cc2314f9beb5b","size":41265},……{"oid":"1c356214ed987a424fb9a713c26a8287469cf35f4d369ac72cd9a5565dc2c815","size":25820}],"ref":{"name":"refs/heads/master"}}02:20:40.221400 trace git-lfs: HTTP: 404
-
误打误撞,根据
external filter ‘git-lfs filter-process’ failed … smudge filter lfs failed on Mac #3519
EOF and exit status 255 connection closed by remote host for 1 user in a specific location #2271
猜测可能是docker网络的某种原因,所以限制:
# lfs.tlstimeout=60 (default 30 seconds)
# lfs.concurrenttransfers=1 (default 3)
git config --global lfs.tlstimeout 60
git config --global lfs.concurrenttransfers 1
git config --global lfs.activitytimeout 60
git config --global lfs.transfer.maxretries 10
git config --global lfs.transfer.maxverifies 8lfs.tlstimeout=60 (default 30 seconds)
lfs.concurrenttransfers=1 (default 3)
克隆成功
-
也许是速率的原因:
来自:API rate limit exceeded
Closing this out. API rate limit has not changed. Unauthenticated rate limits across all of GitHub’s API have been fairly low historically. Configuring a password or turning lfs.access
to basic as I demonstrated in #2134 (comment) will give you the higher, authenticated rate limit.
来自: “API rate limit exceeded” not considered an error
Bah, I left out one important detail. If you set lfs.<url>.access = basic
, you’ll need a login too. You can create a github personal access token with no scopes. This will get you a higher rate limit, without providing access to any private resources.
-
也有可能是内部ip原因:
来自: API rate limit exceeded
@jonasmalacofilho We also saw this exact same problem at the same time you did with the same IP ( 172.16.40.2) address reported in the error messages. It brought all of our work to a halt and then resolved on its own. 172.16.* is an address block reserved for private network routing, so I believe this was probably caused by an internal error at github or one of their providers rather than being due to an actual increase in api requests.
- Git LFS
- Git LFS 操作指南
- 关于存储和带宽使用情况
- Git LFS 服务器实现杂谈
- Git LFS Specification
- external filter ‘git-lfs filter-process’ failed … smudge filter lfs failed on Mac #3519
- EOF and exit status 255 connection closed by remote host for 1 user in a specific location #2271
- API rate limit exceeded
- “API rate limit exceeded” not considered an error
Git LFS文章目录Git LFS1. What is Git LFS?1.1 Creating a new Git LFS repository1.2 Cloning an existing Git LFS repository1.3 Speeding up clones1.4 Pulling and checking out1.5 Speeding up pulls1.6 Tracking...
Git LFS入门微型站点。
本地克隆: git clone https://github.com/git-lfs/git-lfs.github.com.git
运行Jekyll: script/server
在本地浏览器中查看:
生成站点的代码(不包括资产,内容和布局目录的所有内容)以及站点上的代码示例均根据CC0-1.0进行许可。 CC0放弃所有版权限制,但不授予您任何商标许可。
站点内容(资产,内容和布局目录中的所有内容,不包括单独标记的开源许可下的文件)均根据CC-BY-4.0进行了许可。 CC-BY-4.0授予您几乎可以出于任何目的使用内容的权限,但不授予您任何商标权限,只要您注意许可并给予信誉即可,例如:
基于CC-BY-4.0许可使用的基于git-lfs.github.com的内容。
这意味着您可以在自己的
同样是记录一下自己工作遇到的问题,免得下次再遇到了还到处网上查资料解决。
自己的项目的版本控制用的是Git,代码仓库在github托管。项目里用到了百度导航SDK,由于百度导航SDK有了新版本,于是就更新到了新版本,更新好了之后想把代码push到github上,结果出错了,被拒绝,具体信息是:
Total 3007 (delta 664), reused 0 (delta 0)
remote...
什么是Git LFS
git是程序员开发程序不可或缺的工具,有效的使用git能够极大的加快程序人员的开发效率。
在开发比较轻量化的代码时,开发的速度不会受到git上传下载速度的影响,但是随着系统的复杂度增加,代码中关联到的文件越来越多,其中二进制文件发生变化时,git需要存储每次提交的变动,导致本地git仓库越来越大,上传下载速度也受到了很大影响。
Git LFS的出现解决了这一问题,LFS全称Large File Storge,即大文件存储,可以帮助我们管理比较大的文件,对于二进制文件来说,git lf
如果我们用git管理的项目中出现了一些大文件, 同时若其数量比较多, 而且更新又比较频繁, 那么当首次clone该项目时, 就会不可避免地将这些大文件的当前版本和历史所有版本的文件都下载下来, 虽然你很可能用不到这些历史文件, 但是却不得不为它们所占用的额外空间买单.
**git lfs**(large file storage)就很好地解决了这一问题. 通过将大文件替换为小指针的方式, 当你真正需要到这些大文件的时候, 它才会从本地或者远端的lfs缓存中下载这些大文件.
LFS是Large File Storage的缩写,用了帮助git管理大的文件
原理:不同于git每次保存diff,对于git来说,如果是模型或者一些设计大文件,改变一点,对于仓库来说会增加很大的体积,不一会就能几个G。对于git lfs来说,在使用git lfs track命令后,git push的时候,git lfs会截取要管理的大文件,并将其传至git lfs的服务器中,从而减小仓库的体积
注意:git lfs 要求 git >= 1.8.2
1. cent
仗剑过中原:
python3 爬虫新手笔记(二):PRAW、API爬取Reddit
weixin_54791433:
python3 爬虫新手笔记(二):PRAW、API爬取Reddit
Eternity07:
python3 爬虫新手笔记(二):PRAW、API爬取Reddit
python3 爬虫新手笔记(二):PRAW、API爬取Reddit