.pack and .idx file descriptors left open

[jantman]

Sorry, but I believe I found another issue. It appears (maybe I'm missing something here, because this seems very strange to me) that file descriptors are left open after the Repo object is unused or destroyed.

(foo)jantman@jarvis:pts/8:~/tmp/foo$ cat test.py
#!/usr/bin/env python

import git
import commands
import os
import time

pid = os.getpid()

def show_lsof():
    global pid
    cmd = "lsof -p {p} | grep -v /usr/lib".format(p=pid)
    print("=============================")
    print("running {c}:".format(c=cmd))
    print(commands.getoutput(cmd))
    print("=============================")

for rname in ['testrepo', 'testrepo2', 'testrepo3']:
    print("=> Creating repo object for {rname}:".format(rname=rname))
    r = git.Repo(rname)
    show_lsof()

    print("=> iterating commits")
    for c in r.iter_commits():
        pass
    show_lsof()

    time.sleep(2)
    print("=> del r")
    del r
    time.sleep(2)

    show_lsof()
(foo)jantman@jarvis:pts/8:~/tmp/foo$ ./test.py  
=> Creating repo object for testrepo:
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  88651 pipe
=============================
=> iterating commits
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  91431 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> del r
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  91434 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> Creating repo object for testrepo2:
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  91435 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> iterating commits
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  87601 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    6r   REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    7r   REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> del r
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  90605 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    6r   REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    7r   REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> Creating repo object for testrepo3:
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  87612 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    6r   REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    7r   REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> iterating commits
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  79855 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  79854 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  87616 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    6r   REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    7r   REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    8r   REG   0,30    42220  79854 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    9r   REG   0,30     3032  79855 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================
=> del r
=============================
running lsof -p 5325 | grep -v /usr/lib:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
python  5325 jantman  cwd    DIR   0,30      200  73513 /tmp/jantman/foo
python  5325 jantman  rtd    DIR    8,5     4096      2 /
python  5325 jantman  txt    REG   0,30     6240  73547 /tmp/jantman/foo/bin/python2
python  5325 jantman  mem    REG   0,30    33020  73600 /tmp/jantman/foo/lib/python2.7/site-packages/gitdb/_perf.so
python  5325 jantman  mem    REG   0,30     3032  79855 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  79854 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman  mem    REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman  mem    REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    0u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    1u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    2u   CHR  136,8      0t0     11 /dev/pts/8
python  5325 jantman    3r  FIFO    0,8      0t0  88654 pipe
python  5325 jantman    4r   REG   0,30    42220  83920 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    5r   REG   0,30     3032  83921 /tmp/jantman/foo/testrepo/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    6r   REG   0,30    42220  89333 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    7r   REG   0,30     3032  89334 /tmp/jantman/foo/testrepo2/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
python  5325 jantman    8r   REG   0,30    42220  79854 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.pack
python  5325 jantman    9r   REG   0,30     3032  79855 /tmp/jantman/foo/testrepo3/.git/objects/pack/pack-bfea6cfc8edeb84afcc3c9f4d36ce56ed8945db3.idx
=============================

The same thing happens if I explicitly r = None. This is a real problem for me, as I'm trying to write an analysis script that looks at commits in about 450 repositories. On Linux with a default open file handle limit of 1024, the script ends up crashing as every repository I examine results in open file handles for the pack and index files, that appear to never be cleaned up.

I feel like I have to be doing something wrong here, as I don't understand why these aren't cleaned up even after I'm done using the Repo object, and I set it to None or del it.

[Byron]

I think this one might be very close to issue #287, which also provides a workaround: You can use a different back-end which will not leak (at least not in the python process):

import git
r = git.Repo(odbt=git.GitCmdObjectDB)

Also I don't think I can fix this permanently, except for maybe changing the default object database type to the one mentioned. The reason for me being so pessimistic about it is that I came to believe destructors in python cannot be trusted, especially in python 3. Maybe there is some sort of cycle that prevents cleanup in python 2.

Please let me know what you think.

[jantman]

Ok. It's working OK now with the GitCmdObjectDB backend. I couldn't easily find the code that actually opens these files, but it seems to me that perhaps using a context manager would help? Or, since this is an edge case, maybe rather than relying on destructors, this could be handled using an explicit .close_db() method on the Repo object?

[Byron]

Good to hear.
I think the open handles problem originates in smmap, which memory maps everything and provides windowed access to the mapped regions. It is made to de-allocate unused regions when a certain amount of max memory is used. If I remember correctly, it will also deallocate in case it can't open files anymore due to OS limits. However, that functionality was never tested and probably doesn't work.

Everything is possible, yet I am in bare-minimum maintenance mode which means that workarounds like the one you use will be all I can do in this case.
Thanks for your understanding.

[FrostyX]

I've spent two days on this and I have no idea how to fix it. 💩 :guilty: ...

I dig down through the code of GitPython, gitdb and then smmap and in my opinion the true issue lays down in the smmap project. Fixing it above would be just workaround I think.

Please look at the following code, which is based on PackFile::_set_cache_(...):

from smmap import SlidingWindowMapManager
packpath = "/.../.git/objects/pack/pack-022615213253184faeb810dbff2eb8585786d874.pack"
mman = SlidingWindowMapManager()
x = mman.make_cursor(packpath).use_region()

This is place, where those descriptors appears (because of use_region()). We expect that it properly ends, because there is some cleaning in __del__ method. However, any of these will not do the cleaning up:

x.unuse_region()
del x

I've found two ways how to do it, but one which is kinda violent and second one I couldn't manage to pass tests, so there will probably be something stinky about it too.

Firstly, deleting the mman variable definitely helps. Problem here is that it is defined on module-level so deleting it will be, lets say, not particularly shiny.

Secondly, there is method unuse_region() on WindowCursor in which is line self._region = None. I think that exactly this is the broken place. This line in fact throws out the current reference to MapRegion object so we don't have an opportunity to access it again, but it still exists and holds the descriptors. Try to put self._region.release() above the line (to the condition, cause _region can be None) and my example code will work. Unfortunately it breaks some other stuff which I can't understand. Even so I think that deleting WindowCursor should free the descriptors.

I hope this post will spare some time to the person who will try to fix it.

[Byron]

Thank you for posting your findings !
Also I am very sorry to hear you spent so much time on this ... really ... nobody should have to do that as the code was built with the false premises of having working (i.e. deterministic) destructors :( !

Also I hope you are aware of the fix using the GitCmdObjectDB as odbt when instantiating Repo objects - this solved the problem for jantman, and is the only workaround I know.

[jantman]

For me at least, I think I'd be happy with an update to the docs somewhere to mention the workaround. I understand being in maintenance mode, and I'm probably never going to have to deal with repos that big again (I hope)...

[Byron]

In order to fix it once and for all, the default object database type is now one using the system's git command. Additionally, the usage of the pure-python GitDB type is now discouraged.

Even though that won't fix this issue, it's the best I can do. If you see anything else that could be done here, please let me know, and feel free to reopen the issue.

[Byron]

You can watch the development stream on youtube.

GitPython #21 [issue 304 - fix resource leaks once and for all]