Gitlab::GitalyClient::ServerService.new("default").storage_disk_statistics# For Gitaly ClusterGitlab::GitalyClient::ServerService.new("<storage name>").disk_statistics
Gitaly uses the
gRPC
RPC framework. The Ruby gRPC
client has its own log file which may contain helpful information when
you are seeing Gitaly errors. You can control the log level of the
gRPC client with the
GRPC_LOG_LEVEL
environment variable. The
default level is
WARN
.
To fix this problem, confirm that your
gitlab-secrets.json
file
on the Gitaly server matches the one on Gitaly client. If it doesn’t match,
update the secrets file on the Gitaly server to match the Gitaly client, then
reconfigure
.
If you’ve confirmed that your
gitlab-secrets.json
file is the same on all Gitaly servers and clients,
the application might be fetching this secret from a different file. Your Gitaly server’s
config.toml file
indicates the secrets file in use.
If that setting is missing, GitLab defaults to using
.gitlab_shell_secret
under
/opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret
.
Repository pushes fail with a
deny updating a hidden ref
error
Due to
a change
introduced in GitLab 13.12, Gitaly has read-only, internal GitLab references that users are not
permitted to update. If you attempt to update internal references with
git push --mirror
, Git
returns the rejection error,
deny updating a hidden ref
.
The following references are read-only:
refs/environments/
refs/keep-around/
refs/merge-requests/
refs/pipelines/
To mirror-push branches and tags only, and avoid attempting to mirror-push protected refs, run:
Because of changes
introduced
in GitLab 14.10, applying the
noexec
option to a mount
point (for example,
/var
) causes Gitaly to throw
permission denied
errors related to forking processes. For example:
For offline environments where access to public
pool.ntp.org
servers is not possible, the Praefect
check
sub-command fails this
check with an error message similar to:
checking with NTP service at and allowed clock drift 60000ms [correlation_id: <XXX>]
Failed (fatal) error: gitaly node at tcp://[gitlab.example-instance.com]:8075: rpc error: code = DeadlineExceeded desc = context deadline exceeded
To resolve this issue, set an environment variable on all Praefect servers to point to an accessible internal NTP server. For example:
Read distribution caching
is disabled, increasing the number of queries made to the
database when user traffic is high. Ensure read distribution caching is enabled.
With legacy election strategies in GitLab 13.12 and earlier, the primary was the same for all repositories in a virtual storage.
To determine the current primary Gitaly node for a specific virtual storage:
Gitaly Cluster maintains a
metadata database
about the repositories stored on the cluster. Use the
praefect metadata
subcommand
to inspect the metadata for troubleshooting.
You can retrieve a repository’s metadata by its Praefect-assigned repository ID:
Name of the Gitaly storage that contains the replica.
Assigned
Indicates whether the replica is expected to exist in the storage. Can be
false
if a Gitaly node is removed from the cluster or if the storage contains an extra copy after the repository’s replication factor was decreased.
Generation
Latest confirmed generation of the replica. It indicates:
- The replica is fully up to date if the generation matches the repository’s generation.
- The replica is outdated if the replica’s generation is less than the repository’s generation.
-
replica not yet created
if the replica does not yet exist at all on the storage.
Healthy
Indicates whether the Gitaly node that is hosting this replica is considered healthy by the consensus of Praefect nodes.
Valid Primary
Indicates whether the replica is fit to serve as the primary node. If the repository’s primary is not a valid primary, a failover occurs on the next write to the repository if there is another replica that is a valid primary. A replica is a valid primary if:
- It is stored on a healthy Gitaly node.
- It is fully up to date.
- It is not targeted by a pending deletion job from decreasing replication factor.
- It is assigned.
Verified At
Indicates last successful verification of the replica by the
verification worker
. If the replica has not yet been verified,
unverified
is displayed in place of the last successful verification time. Introduced in GitLab 15.0.
Is
some cases
the Praefect database can get out of sync with the underlying Gitaly nodes. To check that
a given repository is fully synced on all nodes, run the
gitlab:praefect:replicas
Rake task
that checksums the repository on all Gitaly nodes.
The
Praefect
dataloss
command only checks the state of the repository in the Praefect database, and cannot
be relied to detect sync problems in this scenario.
View pricing
to see all GitLab tiers and features, or to upgrade. Try GitLab for free
with access to all features for 30 days.
Get Help
If you didn't find what you were looking for,
search the docs.
If you want help with something specific and could use community support,
post on the GitLab forum.
For problems setting up or using this feature (depending on your GitLab
subscription).