Output as absolute distance data in floating point format

[aulerius]

Hey!

I came here looking for higher accuracy depth maps for my SD animation needs. Great work! I'm really glad somebody is doing it!

A big step up for professional use would be to have these depth maps as non-normalized data. Is that even possible with how Midas and other models work? If normalization is deliberately done on the conversion to image, then I'd love an option to skip that part and output real, absolute distance values in floating point format, supported by some image formats, such as TIFF or EXR.

I'm looking to use this on frame-by-frame animation, using depth to accurately displace pixels. Having absolute depth would mean more truthful parallax, when different "scales" in scenes are concerned.

Also, as a side note, I'm all ears for API support through A1111 WEBUI. Right now I will be trying to use it through "script_name": "string" with "script_args": [], guessing the arguments, but I'd be nice if that was at least minimally documented, even if it's half working and still being worked out.
I followed some leads on this repo for the standalone endpoint API, looking at examples, but got stuck with errors.

[graemeniedermayer]

I don't believe calibrated depth is possible with Midas because of the training sources. Zoedepth does have better temporal consistency although it still isn't amazing.

In code, the depthmap is divided by max possible value not normalized.

stable-diffusion-webui-depthmap-script/src/core.py

Line 160 in b8120b4

out /= 2.0 ** bit_depth

A nearby comment appears to address your problem.

# Trying desperately to rescale image to [0;1) without actually normalizing it
# Normalizing is avoided, because we want to preserve the scale of the original depthmaps
# (batch mode, video mode).

Temporal depth consistency for animations probably requires some newer methods.

[aulerius]

I see! I'm not too concerned with temporal consistency though.

So do I get it right that the output is in fact never truly normalized, at least not in the code of this repo? And this applies to any model?

However, If I understand it correctly, If out.max() is the highest pixel value from the depthmap result, then using that to determine bit depth only acts to choose the right amount of bit precision for the output file. The operation indirectly ends up confining the output, whereas a true floating point output can have arbitrary pixel values, even in hundreds or thousands.

Basically, the furthest point in an indoor scene, and a furthest point in an outdoor scene generally looks the same across generations in conventional setups. With floating point values, the outputs should more truthfully reflect the difference in scale between these 2 cases (if the models even care enough to not normalize it in their internal behavior)

[semjon00]

Exactly as you said - the outputs are normalized to [0; 2**16)

[graemeniedermayer]

you might still run into issues with the original training sets. Zoedepth includes some information on their training data.

Zoedepth ranges are outdoors [0.001m, 80m] and indoor [0.001m, 10m].

https://github.com/isl-org/ZoeDepth/blob/main/zoedepth/utils/config.py

I wonder if you can just use the training range to recalibrate the images?

[semjon00]

@aulerius Good idea! Hope someone takes up this task... 🙄🙃

[aulerius]

If you're looking at me, I'm an artist (student of your city in fact 💙), amateur coder, trying to cobble up my workflow from many different parts. I hardly find my way around foreign code like this, but I could maybe test exporting to such image formats on my own edited code.

[semjon00]

No worries, starting with programming can be quite challenging - some people (e.g. me) say it is exciting and that challenges are usually fun - but it might take a lot of time and effort, sometimes uphill battles. But if you decide to try - have fun and edu sulle (good luck)! :)