I’m almost 99% sure the problem is not MKVToolNix, but I’m hoping that maybe someone has had this issue and solved it.
When muxing a 20GB file, the muxing will start off normally and get up to 22%, then hang for a few seconds and then will pick up but go very slowly. That hang time varies depending on the size of the file.
I am completely baffled by this and don’t have an idea of where to begin.
I’m using a new build with an Asus Maximus Z690 Formula, i9-13900K, MSI RTX-3060, 64 GB DDR5 memory and Windows 11 Pro.
Maybe if someone can point to what MKVToolNix uses the most in resources I may have a starting point.
I’ve run memtest several times, and all sorts of other diagnostics and nothing stands out. I don’t see a great deal of CPU or RAM usage.
Thanks.
Welcome!
There are several reason why this happens: read caching (the OS might have parts of the file cached but not everything), write caching (the OS might buffer a lot of the data mkvmerge writes before it has to actually write it to the drive, at which point performance will go down a lot), file content.
Does the mkvmerge process rapidly grow in size during that slower period? I’m not talking about going up 10 MB in a minute, but rather it growing way beyond 1 GB in memory size.
Thanks for the reply.
I don’t know the answer but I will try to figure it out.
If I have it straight…I should monitor the mkvmerge process when it “pauses”, and NOT the CPU, or anything else?
I’ll try it right away.
When you say the OS buffers data, what exactly does this mean? RAM? I only ask because I’m using DDR5, and I’ve been wondering if that’s the issue - memetest always tests OK.
It really is strange, and I’ve been playing with computers since the Commodore computers. This thing has me very confused.
If you’re on Windows, simply use the task manager. Start multiplexing, click in the mkvmerge.exe process so it stays highlighted, and observe its memory usage. It’s totally normal that the usage grows over time, but it should grow slowly, definitely not above 1 GB. The latter would indicate an issue with the content.
You can also take a look at CPU usage & disk I/O usage. High disk I/O would indicate that the write operations are likely the bottleneck (mkvmerge is usually limited by I/O speed, either of memory or of disks).
When you say the OS buffers data, what exactly does this mean? RAM?
I’m not talking about defective memory. What I mean is: when a program opens a file & reads from it the operating system will not only deliver the data to the program but also store it in temporary memory regions. If the same data is read again (by the same program or other programs) the operating system doesn’t have to go to the disk immediately if it still has a copy of that data in that temporary memory region (call the “read cache” or generally the “disk cache” or something similar).
The other way can also benefit from caching: when a program tries to write to a file the data is usually first stored by the operating system in the same “disk cache” memory region instead of going directly to disk. When further write operations to the same file are made they’re also stored in the “disk cache”. All that data can then be grouped into fewer but bigger actual write operations to the disk. This may happen quite a number of seconds after the program made the write requests. When totally depends on the operation system.
Now imagine mkvmerge reading 4 GB of data (roughly 20% of 20 GB) from the source files. This will be handed to mkvmerge but also stored in the cache. mkvmerge will also try to write the same amount to the output file. Those other 4 GB will also be stored in the disk cache first. If all those reads & writes are handled solely by the cache, things will be very fast.
However, at some point the operating system will have to do actual reads from the disk when mkvmerge requests parts of the file that aren’t currently cached. This will be much, much slower. Similarly at some point the operating system will have to write the cached content to the disk, and again, that is very, very slow. At that point the speed will decrease significantly & usually stay low.
All of that is very normal. I’m definitely not suggesting you have defective hardware, nor do I think that you do.
A couple of questions:
When you say “disk cache” - do you mean that the data is NOT stored on RAM but on the C: drive?
What I get from this is that somehow my cache has become limited…?
A bit more info. The file I’m muxing is in my H: drive, and it is an HD drive, my C: drive is an SSD. Also, I’m reading, and writing to the same drive - I have been doing this for a bit, prior to this new build, and I never had this issue.
What’s confusing the heck out of me is that I can’t point to one single thing that fails testing.
I’ll do what you suggested and monitor the process, and maybe that will reveal some weakness in my system.
Thanks for the reply.
The data is ultimately stored on the disk, e.g. the C: drive. However, the operating system keeps copies of some data in memory in order to speed up often-accessed data.
Caching is a rather general computer concept. You can find tons of in-depth explanation online how it works & what the benefits are. All operating systems cache; it isn’t something you have to actively enable.
It’s highly unlikely that there’s a real weakness in your system. This behavior is simply the way things are.
“It’s totally normal that the usage grows over time, but it should grow slowly, definitely not above 1 GB. The latter would indicate an issue with the content.”
I don’t think it’s the content - this happens all the time, with all sorts of content.
I’m just about to try what you suggested, but I think that you have defined the problem very well: “However, at some point the operating system will have to do actual reads from the disk when mkvmerge requests parts of the file that aren’t currently cached.”
What mystifies me is that in my previous build, an old asus MB, old CPU…etc, I never had this problem - the remux went very quickly and had zero pauses.
I would think that with everything upgraded, everything would be better, but it seems not.
I’m going to try what you suggested right now, and I’ll let you know what I see. I’ll monitor mkvmerge, and all the I/Os, as well as hard drive usage.
Sorry about being confused - I have never had this sort of problem before.
Thanks again for the help.
Well, I just tried a 30GB file. CPU was at 1%, memory was at 6%, and the HD I was reading from & writing to was at 100% - this is the same way I’ve done it before…without that freeze. I didn’t manage to catch that point where the process pauses, and I suppose I’ll try it again tomorrow.
At least now I know the CPU & RAM is not the problem. Worst case, I’ll reinstall Windows 11 and see what happens.
Thanks for all the help.
Well, I finally noticed something.
The remux was going very speedily until the hard disk usage went to 98% and then it stopped for a few seconds, and then picked up again but much slower.
I would have thought the system would go to RAM at that point, but evidently not.
I must have changed something because I do NOT remember ever seeing this behaviour.
Thanks for all the help.