I am using mkvmerge for over a decade and wanted to improve the processing time.
My video files are TS files. I use ccextractor to extract the teletext subtitles and merge/add them into an mkv.
The extracted .srt file can contain conversation colors like:
This results into two subtitle streams.
Inspecting them shows that the first subtitle stream, converted from the source file does not contain any font colors, the second (from the .srt) does contain font colors.
As mkvmerge can read teletext subtitles (in a similar way as ccextractor), why does it miss these font colors? Am I missing a setting?
At the moment mkvmerge simply doesn’t handle colors in teletext pages at all. It only handles the text portion. I don’t think I’m going to implement color handling as teletext is a very arcane format that I really hated having to deal with.
After thinking a bit about it, I might look into adding support for colors. It’d help to have a sample file. Can you please upload your file (or at least a part of it that contains color changes) to my file server, please? Thanks!
Thanks! I cannot give you a guarantee or an ETA for an implementation, only that I’ll look into it and decide whether or not to go forward based on how complex the specs look.
I have actually looked into what it would takes to support color in teletext subtitles, and unfortunately it’s a LOT of work. This is ancient technology from decades ago, and hence has been implemented with limitations of TVs & analog signal transmission in mind. In short, this is nothing I’m going to tackle soon, if ever. Sorry.
I had a look two weeks ago into ccextractor, the parser of the character stream just identifies the colors which are less <32, besides the characters and has some logic to keep track of color changes.
What I couldn’t find in mkvmerge where the code is for the transformation of the teletext subtitles to S_TEXT/UTF8. Can you please point me to the source.
Are you using any third-party library for the transformation?
Teletext subtitles can be positioned everywhere on the screen, S_TEXT only where the client positions them. Does mkv support to contain teletext tracks/streams, a "“copy” (couldn’t find a reference for supported subtitle tracks). I understand that clients need to support it too.
mkvmerge converts teletext subtitles to SRT ones in the teletext_to_srt_packet_converter_c
class. It’s my own code; no third-party libraries are involved. The entrypoint is the convert() function which gets called with the assembled PES packet’s payload from the MPEG transport stream (or rather, a data structure containing timestamps/duration as well as a data memer, which contains the PES packet’s payload, meaning no MPEG TS or PES headers, just the codec-relevant data).
I had also looked at ccextractor, but it’s absolutely possible that I got lost somewhere & thought it would be more difficult than I thought.
During conversion mkvmerge simply removes any leading & trailing whitespace, both vertically & horizontally — meaning the all empty lines at the top & bottom are skipped, as are empty characters at the beginning & at the end of each remaining line. What remains is the text that’s kept (after character encoding conversion).
I’d very much appreciate some help here, either in the form of patches/merge requests, but any concrete pointers to the code tracking the colors would be appreciated, too.
I am using QT5 for mythTV when compiling from source.
mkvtoolnix$ ./configure
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking whether gcc and cc understand -c and -o together... yes
checking how to run the C preprocessor... gcc -E
checking for g++... g++
checking whether the compiler supports GNU C++... yes
checking whether g++ accepts -g... yes
checking for g++ option to enable C++11 features... none needed
checking how to run the C++ preprocessor... g++ -E
checking whether make sets $(MAKE)... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking for ranlib... ranlib
checking for strip... strip
checking for ar... ar
checking for ld... ld
checking for strings... strings
checking for objdump... objdump
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for inttypes.h... (cached) yes
checking for stdint.h... (cached) yes
checking for sys/types.h... (cached) yes
checking for sys/syscall.h... yes
checking for stropts.h... no
checking for syscall... yes
checking compiler type and version... gcc 11.3.0
checking for lld... no
checking for support for the "-std=c++17" flag... -std=c++17
checking for support for C++17 feature "attribute 'maybe_unused'"... yes
checking for support for C++17 feature "nested namespace definition"... yes
checking for support for C++17 feature "structured bindings"... yes
checking for support for C++17 feature "std::optional"... yes
checking for support for C++17 feature "std::gcd"... yes
checking for support for C++17 feature "constexpr if"... yes
checking for libraries to link against for the file system library... -lstdc++fs
checking whether the byte order is big-endian... no
checking if being compiled with mingw32... no
checking for int64_t... yes
checking for uint64_t... yes
checking size of int... 4
checking size of long... 8
checking size of long long... 8
checking for iconv.h... yes
checking for iconv... yes
checking for iconv declaration...
extern size_t iconv (iconv_t cd, char * *inbuf, size_t *inbytesleft, char * *outbuf, size_t *outbytesleft);
checking for nl_langinfo... yes
checking for ogg_sync_init in -logg... yes
checking for ogg/ogg.h... yes
checking for vorbis_synthesis_init in -lvorbis... yes
checking for vorbis/codec.h... yes
checking for flac... yes
checking for FLAC__stream_decoder_skip_single_frame in -lFLAC... yes
checking for EBML... no
checking for MATROSKA... no
checking for pugixml.hpp... no
configure: Using the internal version of the pugixml library
checking nlohmann's json-cpp... no
configure: Using the internal version of nlohmann json-cpp
checking for utf8.h... no
configure: Using the internal version of UTF8-CPP
checking fmt... no
configure: Using the internal version of fmt
checking for ZLIB... yes
checking for qmake6... no
checking for qmake-qt6... no
checking for qt6-qmake... no
checking for Qt 6... no: qmake6 not found
checking for qmake-qt5... no
checking for qmake... /usr/bin/qmake
checking for qmake's version... 5.15.3
checking for lconvert... /usr/bin/lconvert
checking for moc... /usr/lib/qt5/bin/moc
checking for rcc... /usr/lib/qt5/bin/rcc
checking for uic... /usr/lib/qt5/bin/uic
checking for Qt 5... no: not found by pkg-config
configure: error: The Qt library is required for building MKVToolNix.
dpkg -l |grep qt
ii libqt5concurrent5:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 concurrent module
ii libqt5core5a:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 core module
ii libqt5dbus5:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 D-Bus module
ii libqt5gui5:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 GUI module
ii libqt5network5:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 network module
ii libqt5opengl5:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 OpenGL module
ii libqt5opengl5-dev:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 OpenGL library development files
ii libqt5positioning5:amd64 5.15.3+dfsg-3 amd64 Qt Positioning module
ii libqt5printsupport5:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 print support module
ii libqt5qml5:amd64 5.15.3+dfsg-1 amd64 Qt 5 QML module
ii libqt5qmlmodels5:amd64 5.15.3+dfsg-1 amd64 Qt 5 QML Models library
ii libqt5qmlworkerscript5:amd64 5.15.3+dfsg-1 amd64 Qt 5 QML Worker Script library
ii libqt5quick5:amd64 5.15.3+dfsg-1 amd64 Qt 5 Quick library
ii libqt5quickparticles5:amd64 5.15.3+dfsg-1 amd64 Qt 5 Quick particles module
ii libqt5quickshapes5:amd64 5.15.3+dfsg-1 amd64 Qt 5 Quick Shapes module
ii libqt5quicktest5:amd64 5.15.3+dfsg-1 amd64 Qt 5 Quick Test library
ii libqt5quickwidgets5:amd64 5.15.3+dfsg-1 amd64 Qt 5 Quick Widgets library
ii libqt5script5:amd64 5.15.3+dfsg-1 amd64 Qt 5 script module
ii libqt5scripttools5:amd64 5.15.3+dfsg-1 amd64 Qt 5 script tools module
ii libqt5sensors5:amd64 5.15.3-1 amd64 Qt Sensors module
ii libqt5sql5:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 SQL module
ii libqt5sql5-mysql:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 MySQL database driver
ii libqt5svg5:amd64 5.15.3-1 amd64 Qt 5 SVG module
ii libqt5test5:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 test module
ii libqt5webchannel5:amd64 5.15.3-1 amd64 Web communication library for Qt
ii libqt5webkit5:amd64 5.212.0~alpha4-15ubuntu1 amd64 Web content engine library for Qt
ii libqt5webkit5-dev:amd64 5.212.0~alpha4-15ubuntu1 amd64 Web content engine library for Qt - development files
ii libqt5widgets5:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 widgets module
ii libqt5xml5:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 XML module
ii libqt6core6:amd64 6.2.4+dfsg-2ubuntu1.1 amd64 Qt 6 core module
ii libqt6dbus6:amd64 6.2.4+dfsg-2ubuntu1.1 amd64 Qt 6 D-Bus module
ii libqt6gui6:amd64 6.2.4+dfsg-2ubuntu1.1 amd64 Qt 6 GUI module
ii libqt6multimedia6:amd64 6.2.4-1 amd64 Qt 6 Multimedia library
ii libqt6network6:amd64 6.2.4+dfsg-2ubuntu1.1 amd64 Qt 6 network module
ii libqt6opengl6:amd64 6.2.4+dfsg-2ubuntu1.1 amd64 Qt 6 OpenGL module
ii libqt6svg6:amd64 6.2.4-1ubuntu1 amd64 Qt 6 SVG library
ii libqt6widgets6:amd64 6.2.4+dfsg-2ubuntu1.1 amd64 Qt 6 widgets module
ii qt5-gtk-platformtheme:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 GTK+ 3 platform theme
ii qt5-image-formats-plugins:amd64 5.15.3-1 amd64 Qt 5 Image Formats module
ii qt5-qmake:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 qmake Makefile generator tool
ii qt5-qmake-bin 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 qmake Makefile generator tool — binary file
ii qt5-qmltooling-plugins:amd64 5.15.3+dfsg-1 amd64 Qt 5 qmltooling plugins
ii qt6-gtk-platformtheme:amd64 6.2.4+dfsg-2ubuntu1.1 amd64 Qt 6 GTK+ 3 platform theme
ii qt6-qpa-plugins:amd64 6.2.4+dfsg-2ubuntu1.1 amd64 Qt 6 EGL plugins
ii qtbase5-dev:amd64 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 base development files
ii qtbase5-dev-tools 5.15.3+dfsg-2ubuntu0.2 amd64 Qt 5 base development programs
ii qtchooser 66-2build1 amd64 Wrapper to select between Qt development binary versions
ii qtdeclarative5-dev:amd64 5.15.3+dfsg-1 amd64 Qt 5 declarative development files
ii qtdeclarative5-dev-tools 5.15.3+dfsg-1 amd64 Qt 5 declarative development programs
ii qtscript5-dev:amd64 5.15.3+dfsg-1 amd64 Qt 5 script development files
ii qttranslations5-l10n 5.15.3-1 all translations for Qt 5
There are instructions for compiling the DEB package that should work on both Debian & Ubuntu. Install all the requirements as listed there. Afterwards you should be able to run configure without further arguments, and it should find everything. If not, please post the output of configure again & upload the generated config.log. Thanks.
I used the “apt install” from your documentation. Still needed “apt install libboost-system1.74.0”.
For configure I needed to symlink the libraries:
That’s possible; I don’t exactly keep the instructions up to date, and due to how often dependencies change I cannot really provide one set of packages that works across all releases of Debian/Ubuntu/Mint.
What’s more up to date is the Dockerfile I use to create the containers in which all of my provided packages are built. It’s a template file. The important sections for Ubuntu are lines 101–109. Those packages suffice to build MKVToolNix.
You 100% do not need to symlink any of the libraries anywhere for configure to find them on a supported version of Ubuntu (22.04 or newer at the moment).
Boost’s filesystem library uses Boost’s system library under the hood. The other check is for Boost’s operators library which is a header-only library, meaning there’s no shared library to link against. The code’s fully compiled into the executables.
There’s no grand list of options. What you must look for is the initialization of the various debugging_option_c instances. The constructor takes a string argument that contains one or more potential names that turn on that specific option. For example, debugging_option_c m_debug{"moo|moo_full"}; can be turned on with --debug moo or --debug moo_full. You can combine several debugging options with commas.
I’ve seen it. Thank you very much! Looks very easy. I’m really confused what code I was actually looking at that I thought was very complicated. It’s possible I was actually looking at code for closed captions or other similar systems, but not at the code for teletext subtitles.
On first glance your code looks fine. I’ll adjust several things to make it fit into my coding style better, but I won’t make you jump through hoops to change those things yourself. Haven’t given it a try yet & will that tomorrow, I guess.
Debugging “m_debug_packet”:
Still not that straightforward as I needed to use --debug teletext_to_srt_all and not --debug packet.
Forgot to mention, the skipping at the beginning won’t work, as it contains the color attribute (when I am not wrong, it is a required attribute). I also noticed twice that there were two 0xB (11, better show the debug in hex) (in the test file near “Äh…”). Thereafter spaces come.
I added skipping for the leading spaces at the beginning.
Yeah, that’s because packet is not part of the constructor argument to m_debug_packet. --debug teletext_to_srt_all would also have worked, though that it would have triggered other debug output, too.
That should be fixable by a bit of regex postprocessing, e.g. replacing all whitespace directly following <font color=…> or preceding </font>. I’ll look into it.
I use the Boost-specific configure macros from the official autoconf macro archive. They look for the system library first via various names & set environment variables accordingly that subsequent checks for the file system library re-use. I don’t know the actual details, but manually symlinking is 100% not necessary. Those symlinks must exist, or at least they do in my Docker container (see below), but they were created by the regular Ubuntu mechanisms, not manually by myself.
Really not sure. Have you tried installing all the packages listed in the Dockerfile.tt I’ve linked to above?
Maybe as I installed mkvtoolnix the additional libraries were installed with a version number (no symlink).
When libboost-system was missing I installed apt install libboost-system1.74.0.
Besides the versioned package a -dev package exists.
apt-cache search libboost | grep -- -dev
libboost-dev - Boost C++ Libraries development files (default version)
libboost-filesystem-dev - filesystem operations (portable paths, iteration over directories, etc) in C++ (default version)
libboost-filesystem1.74-dev - filesystem operations (portable paths, iteration over directories, etc) in C++
libboost-regex-dev - regular expression library for C++ (default version)
libboost-regex1.74-dev - regular expression library for C++
libboost-system-dev - Operating system (e.g. diagnostics support) library (default version)
libboost-system1.74-dev - Operating system (e.g. diagnostics support) library
libboost-thread-dev - portable C++ multi-threading (default version)
libboost-thread1.74-dev - portable C++ multi-threading
libboost1.74-dev - Boost C++ Libraries development files
libboost1.74-tools-dev - Boost C++ Libraries development tools
libboost-all-dev - Boost C++ Libraries development files (ALL) (default version)
libboost-math-dev - Boost.Math Library development files (default version)
libboost-math1.74-dev - Boost.Math Library development files
libboost-random-dev - Boost Random Number Library (default version)
libboost-random1.74-dev - Boost Random Number Library
libboost1.74-all-dev - Boost C++ Libraries development files (ALL)
<deleted some entries>
I haven’t run ldconfig explicitly (normally no need to do so), but one of the -dev packages made have called it and the symlink’s are created.
Quite possibly the libboost-dev package had a dependency on the other Boost-specific -dev packages at some point in the past. Be that as it may, I’ve updated the web site.