AI depth map estimation for meshroom

[Tom_nerd]

Hi,

I was recently doing a photoscan of a bin using Meshroom, as you do. And was looking at the amount of time spent on depth map estimation for each image. I had around 50 photos although it still took a few hours, granted the system is a little old. I then looked up how Meshroom estimates depth maps, and was baffled by how confusing it was.

I was thinking, in the day and age where AI is everywhere, would it be possible to use an AI to estimate depth maps using AI. I made a model in tensorflow the other day, within hours and was a piece of cake to make. Would it be possible to integrate that instead of the current node in Meshroom? There is a multitude of advantages:
- Would mean that anyone could use mushroom, not just people with a CUDA GPU, people could even run photoscan jobs only on the CPU.
- Would be drastically faster than the current method. Using a Nvidia K80 (inside google colab) I could process each image in less than a second. Much faster than the roughly minute it takes with the current method.
- Would not require downscaling, like the current method. Meaning more data can be fed to the next node.

Please feel free to tell me why this would not work.

Thanks in advance,
Tom.

[pdash]

hi Tom - This is very interesting. I have worked on panoramic image stitching earlier and I had come across a deep learning based feature matching that could match hard features (but SIFT and other matchers missed). It did perform well in some scenarios but not in general. That was back in 2019!! Things definitely have improved in AI space and I can believe AI/Deeplearning based matches would perform better and work on CPU. If you have built an AI model that works and you can match the output that comes out of normal feature-matching based pose-estimation, it should work. I would be very interested to know more about it and explore further.

thanks
Dash

[Micr0]

Hi,

I was recently doing a photoscan of a bin using Meshroom, as you do. And was looking at the amount of time spent on depth map estimation for each image. I had around 50 photos although it still took a few hours, granted the system is a little old. I then looked up how Meshroom estimates depth maps, and was baffled by how confusing it was.

I was thinking, in the day and age where AI is everywhere, would it be possible to use an AI to estimate depth maps using AI. I made a model in tensorflow the other day, within hours and was a piece of cake to make. Would it be possible to integrate that instead of the current node in Meshroom? There is a multitude of advantages:
- Would mean that anyone could use mushroom, not just people with a CUDA GPU, people could even run photoscan jobs only on the CPU.
- Would be drastically faster than the current method. Using a Nvidia K80 (inside google colab) I could process each image in less than a second. Much faster than the roughly minute it takes with the current method.
- Would not require downscaling, like the current method. Meaning more data can be fed to the next node.

Please feel free to tell me why this would not work.

Thanks in advance,
Tom.

Hi, Tom.
This is timely. I too am also interested in using AI in depth mapping, but for a completely different application. I have a friend who uses what amounts to 3D textile weaving machines. The Software used to program the machine is proprietary, but it can import. .tiff image files. Those files can then be used for depth maps to control the weaving heads. I wanted to find a way to use ai. to map a 3D mesh into one of these 2D depth images for use with these machines. I have to admit, I have., I have no understanding how to go about even starting a project like this. Let us know what you find out, it may be helpful.