I cloned a mirror of my remote, ran git-lfs-migrate, git-gc, and attempted to push the mirror to gitlab.com multiple times. On each push attempt, I tried different suggestions found online:

All attempts fatal. On each fatal, the error messages were similar, but varied slightly.

http/1.1 recently failed with http 413, indicating some entity was too large. I stuck with the default core.compression (by being unset), I switched back to http2, and am going to do a more aggressive git gc --prune=now --aggressive . This failed with "pack exceeds maximum allowed size".

I'm going to try to run git gc --prune=now --aggressive but with the following config:

git config core.packedGitLimit 256m
git config core.packedGitWindowSize 256m
git config pack.deltaCacheSize 256m
git config pack.packSizeLimit 256m
git config pack.windowMemory 256m

I think this will repack each pack to a max size of 256m. Then I'll try the push again.

Attempted http/2 with core.compression set to 9 (max). That failed. Going to try the same with http/1.1. I tried http/1.1 in the past, but I don't think I've tried it after a core.compression set to 9 and gc. My current config is:

git config http.postBuffer 134217728
git config core.compression 9
git config core.packedGitLimit 128
git config core.packedGitWindowSize 128
git config pack.deltaCacheSize 128
git config pack.packSizeLimit 128
git config pack.windowMemory 128
git config http.version HTTP/1.1  

which, if I'm understanding things correctly, the max size of any pack file is 128M. Maximum compression is on. We need to gc the repo first to map git objects to smaller pack files if they're too big. Http post buffer is sized to support objects of 128M. We'll use http/1.1. I think that summarizes what I'm going for pretty accurately.

The most recent attempt failed with "error: RPC failed; curl 56 OpenSSL SSL_read: Connection was aborted, errno 10053". I'll paste the fuller error output down below. I'll retry with double the http.postBuffer at 256M, which should be more than enough for these packs of no more than 128M.

I keep getting "RPC failed; curl 56 ..." error, and since the error message keeps suggesting I increase the http.postBuffer, I'll go ahead and double it each time I get this same error message. I'm currently attempting 1G, with the rest of the git-config fixed to the same values.

cd src
git remote set-url origin https://gitlab.com/path/to/src.git

git config http.postBuffer 1073741824
git config core.compression 0
git config http.version HTTP/1.1

Here are those links that suggest these config vars:

The project that I can reproduce this on is an internal/private one. It is also ~30 GB in size. I have not successfully reproduced this behavior on a minimal project.

On git-push, one of, or similar to...

I got the "curl 92" error with HTTP/2 and core.compression 9 (max), following an aggressive gc:

09:14:05.705694 http.c:689              == Info: HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
09:14:05.705694 http.c:689              == Info: stopped the pause stream!
09:14:05.705694 http.c:689              == Info: Connection #0 to host gitlab.com left intact
error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
send-pack: unexpected disconnect while reading sideband packet

Using http/1.1, I get "something's too large":

01:15:10.838058 http.c:781              == Info: HTTP error before end of send, stop sending
01:15:10.838058 http.c:728              <= Recv header, 0000000002 bytes (0x00000002)
01:15:10.838058 http.c:740              <= Recv header:
01:15:10.838058 http.c:781              == Info: Connection #1 to host gitlab.com left intact
error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly

23:04:26.250318 http.c:689              == Info: HTTP error before end of send, stop sending
23:04:26.250318 http.c:636              <= Recv header, 0000000002 bytes (0x00000002)
23:04:26.250318 http.c:648              <= Recv header:
23:04:26.252319 http.c:689              == Info: Connection #0 to host gitlab.com left intact
error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly

16:29:16.915645 http.c:781              == Info: Signaling end of chunked upload via terminating chunk.
16:29:58.175517 http.c:781              == Info: Mark bundle as not supporting multiuse
16:29:58.175517 http.c:728              <= Recv header, 0000000039 bytes (0x00000027)
16:29:58.175517 http.c:740              <= Recv header: HTTP/1.1 413 Request Entity Too Large
16:29:58.175517 http.c:728              <= Recv header, 0000000037 bytes (0x00000025)
16:29:58.175517 http.c:740              <= Recv header: Date: Sat, 22 Jan 2022 00:29:58 GMT
16:29:58.175517 http.c:728              <= Recv header, 0000000025 bytes (0x00000019)
16:29:58.175517 http.c:740              <= Recv header: Content-Type: text/html
16:29:58.175517 http.c:728              <= Recv header, 0000000021 bytes (0x00000015)
16:29:58.175517 http.c:740              <= Recv header: Content-Length: 181
16:29:58.175517 http.c:728              <= Recv header, 0000000024 bytes (0x00000018)
16:29:58.175517 http.c:740              <= Recv header: Connection: keep-alive
16:29:58.175517 http.c:728              <= Recv header, 0000000020 bytes (0x00000014)
16:29:58.175517 http.c:740              <= Recv header: Server: cloudflare
16:29:58.180566 http.c:728              <= Recv header, 0000000030 bytes (0x0000001e)
16:29:58.180566 http.c:740              <= Recv header: CF-RAY: 6d14a28d2b4a7081-SJC
16:29:58.180566 http.c:728              <= Recv header, 0000000002 bytes (0x00000002)
16:29:58.180566 http.c:740              <= Recv header:
16:29:58.181109 http.c:781              == Info: Connection #1 to host gitlab.com left intact
error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly

Got the following after making sure compression is on, did git gc --prune=now --aggressive , http2

04:20:33.256553 pkt-line.c:80           packet:     sideband< \2fatal: pack exceeds maximum allowed size
remote: fatal: pack exceeds maximum allowed size        

With HTTP/1.1, pack size no greater than 128M, postBuffer size 128M. Going to double http post buffer to 256M. With the recent gc however, I think I have guaranteed that no pack exceeds 128M.

21:14:51.141239 http.c:728              => Send header, 0000000329 bytes (0x00000149)
21:14:51.141239 http.c:740              => Send header: POST /nextest-eng/usa/src.git/git-receive-pack HTTP/1.1
21:14:51.141239 http.c:740              => Send header: Host: gitlab.com
21:14:51.141239 http.c:740              => Send header: Authorization: Basic <redacted>
21:14:51.141239 http.c:740              => Send header: User-Agent: git/2.33.0.windows.2
21:14:51.141239 http.c:740              => Send header: Accept-Encoding: deflate, gzip, br, zstd
21:14:51.141239 http.c:740              => Send header: Content-Type: application/x-git-receive-pack-request
21:14:51.141239 http.c:740              => Send header: Accept: application/x-git-receive-pack-result
21:14:51.141239 http.c:740              => Send header: Transfer-Encoding: chunked
21:14:51.141239 http.c:740              => Send header:
21:17:25.895278 http.c:781              == Info: OpenSSL SSL_read: Connection was aborted, errno 10053
21:17:25.895278 http.c:781              == Info: Connection died, retrying a fresh connect (retry count: 1)
error: unable to rewind rpc post data - try increasing http.postBuffer
21:17:25.910939 http.c:781              == Info: the ioctl callback returned 2
21:17:25.910939 http.c:781              == Info: ioctl callback returned error 2
21:17:25.910939 http.c:781              == Info: Closing connection 1
error: RPC failed; curl 56 OpenSSL SSL_read: Connection was aborted, errno 10053
send-pack: unexpected disconnect while reading sideband packet
22:00:38.095300 pkt-line.c:80           packet:          git> 0000
fatal: the remote end hung up unexpectedly

git-push should successfully complete and exit 0

I've redirected the push output and appended each attempt to a ~/push.log . The log is huge. I prefer to provide this as a separate attachment.

This bug happens on GitLab.com

(For installations with omnibus-gitlab package run and paste the output of: sudo gitlab-rake gitlab:check SANITIZE=true )

(For installations from source run and paste the output of: sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true )

(we will only investigate if the tests are passing)