Gswin64c.exe stays alive after Magick.Net thread is finished

[EL-bm]

Hello, hope everybody is doing great.
I have an app with the following features:

• Process multiple files using parallel programming.
  Parallel.Foreach
• Each thread opens up the file, creates an image collection, closes the file stream, and then continues.
  var defines = new PdfReadDefines() { Interpolate = true };
var settings = new MagickReadSettings();
settings.Density = new Density(dpi);
settings.Compression = CompressionMethod.NoCompression;
settings.Format = MagickFormat.Pdf;
settings.StrokeAntiAlias = true;
settings.TextAntiAlias = true;
settings.SetDefines(defines);
using (images = new MagickImageCollection())
{
images.Read(_stream, settings);
_stream.Dispose();
...
}
• When the thread process finishes/fails, the code makes sure the file stream gets properly closed and that the image collection gets cleared and set to null.
  if (_stream != null) { _stream.Dispose(); _stream = null; }
if (images != null) { images.Clear(); images = null; }
• Even with all the dispose mechanisms, the original files gets locked by a process gswin64c.exe (see image 1).

I know Magick.NET is capable of multi-threading and that only one thread will use the dll file and the rest will use the gswin64c.exe as command line.
The question: Is there a way to make sure the Magick.NET instance also kills the gswin64c.exe process it created?
Thanks in advance.

[Image 1]

[dlemstra]

Is it possible for you to create a small sample program (with a PDF) that can reproduce this consistently?

[dlemstra]

You cannot choose a different folder for each execution. Have you checked if the files in the tempFiles folder are properly cleaned up? Maybe the process hangs because you are running out of disk space? It might be possible that those two problem images are super large and forcing MagickNET to swap the pixels to disk. You probably want to limit the max width and height by configuring that in a policy or setting ResourceLimits.Width and Height in runtime.

[EL-bm]

Thanks @dlemstra. Our solution for now is simply avoid using the gswin64c.exe and the way we were able to achieve that is to create copies of the same service and each copy uses a new location of the dll. So we have 4 services per server ,TileCreator1 - 4, and 4 different copies of the dll component so each instance of the services calls a different one. Each server process one file a time so we process 4 files per server (kind of multi-threading but in the hard way).

To answer your questions:
1.- Are we running out of space? -> Yes but the reason is the gswin64c.exe process not dying. Since the exe process persists forever (until the service gets rebooted altogether), the temp files are locked forever. We have a special sub-process in charge of cleaning any temp files that are not being used and that are older than 10 mins.

My apologies to the team since I have had not found the time to write a sample code illustrating this issue mainly because I was forced to look for a quick solution since this is our production process.

• I'll try to find some time to write a sample app.
• Until then it is my opinion there is a problem in the multithreading mechanism of the Magick.NET library BECAUSE it is using the .exe. You guys mentioned in other post that it is not possible to use the dll for all of the threads and I don't really know why.
• Given our results using four different copies of the dll, I strongly believe that this kind of issues would be resolved if Magick.Net could internally use the dll for all the threads.

I don't know if you guys want to mark this thread as closed. Let me know.

[dlemstra]

Having 4 different processes is different from having 4 threads in the same process. The library is not thread safe and that is why the next execution will need to use the binary instead of the library.

Is it possible that the file that causes this issues is corrupt and causes gswin64c.exe to never exit?

[EL-bm]

File corruption is what we review in second place being pdf world a "diverse" one. We looked for incomplete fonts, different types of images and metadata and even custom signatures for bookmarks. What we found is that there is no apparent pattern based on the pdf structure which means the process could fail for a file that would be later process correctly on a second try.
Here is a list of what we checked:

• Antivirus
• Antiramsomware
• PDF corruption
• Network latency and disconnection (our files are located on several network locations)
• Interference by the edition department (a separate group of people works over these same pdf files)
• Interference by our main viewer app (the one that uses the created tiles)
• Tile creator service logic
• Tile creator service interaction with core db (looking for deadlocks that would prevent the proper update of the data causing malfunctioning)
• Magick.NET library

We turned our heads to the Magick.NET library at the end because it showed no issues during development phase. We have been using the Magick.NET library since 2 years ago for creating the tiles but at that time we were using a different component to read a raster version of a PDF. It was until 2 months ago that we started using Magick.Net library combined with ghostscript to also read the raster from the pdf.

After this extensive review we were able to determine that the only common element for most of the issues was the use of the gswin64c.exe. Using the exe will fail for the same file that the dll (or even a different thread using the exe) will process correctly. We don't know why there is such disparity when using the exe not being really experts on ghostscript command line.
We process a volume around 2k - 4k pdf files a day. For our testing we focused on the files we knew were a fail getting a sample space of 44 files. At the end, we were able to replicate the problem using even just 2 files (2 needed to trigger the usage of the binary).

The following questions float on my mind:

• Why the dll is not failing (so far using the dll is not showing issues) and the binary is? This looks like a problem on ghostscript side.
• Why is the binary failing? Even for the same file it'll fail some times and some others won't. Ghostscript issue maybe?
• When the binary is failing, why is the binary execution not getting terminated? This doesn't happen all the time but it is the core issue for us since the original file is getting blocked (preventing other instances to process it) and the temp files are also getting locked eating up the hard drive space eventually leading to the server dying. This looks like on Magick.NET issue.

I am not allowed to share a direct version of our code (NDA) so I am looking for some time to write a code sample so you guys can analyze it but it is a bit hectic being the end of the year and all.

[EL-bm]

Dear @dlemstra ,
Here is a link to download the sample code:
https://www.dropbox.com/s/mrt1roo32wi70pk/TileCreatorSample.zip?dl=0
Sorry for the delay. Please unpack the file and read the instructions.
Hope this helps.

[EL-bm]

In case nobody saw my previous comment from 4 days ago:
Dear @dlemstra ,
Here is a link to download the sample code:
https://www.dropbox.com/s/mrt1roo32wi70pk/TileCreatorSample.zip?dl=0
Sorry for the delay. Please unpack the file and read the instructions.
Hope this helps.

[dlemstra]

Seen your message, haven't had/made time for this yet.

[dlemstra]

I finally made some time to investigate this. It turns out that the process is in a deadlock. We are waiting for the process to exit but the process is waiting for us to read it's output. One of your test files spits out a lot of messages to the output and if I force that file to always use the command line a deadlock will occur. Thanks for providing this example and a fix will become available in the next version of Magick.NET

[EL-bm]

No better way to start 2021 than contribute with others in someway!! I am happy you guys were able to spot this one. I believe it was a tricky error since specific conditions are needed for it to reveal. Let me know if I can be of any assistance.
I'll wait for the next iteration of this awesome library.
You guys have a happy and blessed 2021.
Best regards!

[tomasr78]

I have exactly the same problem - sometimes the produced Image files from PDF are being locked and released after one or two seconds or even never released. I have tried to update to the new Magick.NET 7.22.3.0 version but still, the problem persists. I am using this code to execute pdf to image in parallel #810

I do not find a pattern of how to reproduce the problem, it just happens without a clear reason.

[dlemstra]

Your description does not sound like the exact same problem. PDF files being locked is different from the Ghostscript executable not exiting.

p.s. @EL-bm Can you confirm that this issue is resolved?

[tomasr78]

Ups, sorry, yes the problem may be different and not PDF but created PNG is being locked. I am still trying to debug and find more info.

[EL-bm]

Woops, sorry, been so busy lately that I did not notice there was a new version. I'll try to implement this new version asap. I'll let you guys know.
Thanks for all your hard work.

[EL-bm]

Dear @dlemstra and @tomasr78,
Today we installed the new version in our alpha servers. Everything seems to be working fine. I see the exe app getting disposed off correctly and our app is now running in a single service instance with multithreading.
For a better details, we upgraded our Magick.NET version to the current one: 7.23.0.0.
We'll continue monitoring this new version for a week and a half. I'll report back on Friday (or earlier if any problem) with more details.
Hope this info helps.

[EL-bm]

New update,
The service using the new Magick.Net version (7.23.0.0) got heavily tested yesterday. Thanks to another requirement we got the chance the process a volume over 500k files and I am happy to report that we did not see any of the previous issues.
Being all this said, this bug is fixed in my opinion.
Thank you all for your hard work, this library is amazing in many ways.

[dlemstra]

Happy to hear this got resolved and thank you for helping me fix this.