External subtitles for DVDPlayer (text-based and bitmap-based)

[Gamester17]

Since XBMC's FFmpeg-based in-house DVDPlayer video-player (http://xbmc.org/wiki/?title=DVDPlayer) will be XBMC's primary cross-platform video-player core I like too submit this little more formal suggestion for the handling of external subtitles in the DVDPlayer, preferably via portable (supporting both Xbox/Win32 and Linux/POSIX) library.

Something in the lines what is suggested here in the FFmpeg Wishlist (http://wiki.multimedia.cx/index.php?title=FFmpeg_Wishlist#Subtitles):* Create a common 'subtitles parser library' (and/or an API system for adding support for additional subtitle formats?) - a common sub-library to FFmpeg with all subtile decoders/demuxers/parsers gathered (similar to the libpostproc and libavutils). Call it "libsubs" (or "libsub", "libsubtitles" or whatever). Move FFmpeg's existing VobSub and DVBsub code there, so no matter if they are bitmap or text-based subs all existing and future subtile code is collected there. This will help reduce future code replication by sharing common code, thus making it easier to add support for additional subtitles.
**Maybe use MPlayer's recently added "libass'" (SSA/ASS subtile reader) as a base for such a common library?
* Support for advanced SSA/ASS rendering
**Possible source are libass or the asa library
* Support bold, italic, underline, RGB colors, size changes and font changes for a whole line or part of one line
* Line 23 signal (a.k.a. "Wide-screen signal") (http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=284541) detecting and use for DVD-Video (VobSub)
* Support for the subtitles HTML tags
* Capability of displaying subtitles with no video enabled (for example for audio-books)
* Support for Karaoke subtitles (for kar and cdg, etc.)
* Dual-subtitle-display (display two subtitles/languages at the same time, one at the bottom as normal plus one at the top of the screen)
* Capability of moving the subtitles in the picture (freetype renderer)
*Support more subtitle formats (text and bitmap-based):
** Closed captioning (CC) (http://en.wikipedia.org/wiki/Closed_captioning) subtitle support - (Closed captions for the deaf and hard of hearing, also known as "Line 21 captioning", uses VobSub bitmaps)
*** xine (http://www.xinehq.de) have a SPU decoder for subpictures and Closed Captions software decoding
** DirectVobSub (VSFilter (http://sourceforge.net/projects/guliverkli/)) - standard VobSubs (DVD-Video subtitles) embedded in AVI containers
** DivX Subtitles (XSUB) (http://en.wikipedia.org/wiki/DivX#DivX_Subtitles_.28XSUB.29) display/reader/decoder (Note: bitmap based subtitle, similar to VobSub)
** SubRip (.srt) (http://en.wikipedia.org/wiki/SubRip) subtile support (Note: simple text-based based subtitle with timestamp)
** Subviewer (.sub) subtile support (Note: simple text-based based subtitle with timestamp)
** MicroDVD (.sub) subtile support (Note: simple text-based based subtitle with timestamp
** Sami (.smi) (http://en.wikipedia.org/wiki/Sami_%28subtitle_tool%29) subtile support (Note: simple text-based based subtitle with timestamp)
** SubStation Alpha (.ssa+.ass) (http://en.wikipedia.org/wiki/SubStation_Alpha) subtile support (Note: advanced text-based based subtitle with timestamps and X/Y location on screen)
** RealText (.rt) (http://en.wikipedia.org/wiki/Synchronized_Multimedia_Integration_Language)subti le support
** PowerDivx (.psb) subtile support
** Universal Subtitle Format (.usf) (http://en.wikipedia.org/wiki/Universal_Subtitle_Format) subtitle support
** Structured Subtitle Format (.ssf) subtile supportPossible Source code libraries:

MPlayer (http://www.mplayerhq.hu) (MPlayer has multiple subtitle libaries? like libass for SSA/ASS subtitles)

VSFilter (http://sourceforge.net/projects/guliverkli/) (formally called "VobSub") DirectShow filter (advanced subtitles with styles and realtime karaoke effects. All rendered in high quality with anti-aliased fonts)

[pike]

another possible source to build upon is Guliverkli's subtitle support

[Gamester17]

I wonder how/why FFmpeg added this patch their SVN; Log of /trunk/libavformat/matroskadec.c (http://svn.mplayerhq.hu/ffmpeg/trunk/libavformat/matroskadec.c?sortby=date&view=log)
Revision 9591
Modified Wed Jul 11 13:13:20 2007 UTC (13 days, 1 hour ago) by aurel
add support for Matroska subtitle tracks

Revision 9593
Modified Wed Jul 11 13:37:07 2007 UTC (13 days, 1 hour ago) by aurel
add support for ASS like subtitles in Matroska


Revision 9704
Modified Mon Jul 16 14:51:43 2007 UTC (7 days, 23 hours ago) by aurel
add support for S_ASS and S_SSA codec stringFor more information read their mailling-list conversation => [FFmpeg-devel] [PATCH] matroska subtitle tracks support (http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2007-July/thread.html#32456)

???

Too bad FFmpeg did not add full text-based subtitle support in all supported containers... ...but maybe XBMC devs would be willing to extend their support? Maybe building a genric filter class in FFmpeg's "libavutil" for their 'CODEC_ID_TEXT' to all text-based subtitles?

:rolleyes:

[Gamester17]

another possible source to build upon is Guliverkli's subtitle support@pike, it is Guliverkli who makes VSFilter (http://sourceforge.net/projects/guliverkli/) (formally called "VobSub") which is the DirectShow filter I mentioned in my first post. Problem is since it is made for DirectShow which is Windows Win32 only it would have to be ported to support all platforms, ...on the other hand, VSFilter is codec in C++ and MPlayer is coded in C which I am not sure makes a difference if it would be implemented into XBMC's DVDPlayer?

[Gamester17]

FYI;

FFmpeg (http://ffmpeg.mplayerhq.hu) has now implemented a DivX subtitles (XSUB™) decoder (http://en.wikipedia.org/wiki/DivX#DivX_Subtitles_.28XSUB.29) into their SVN ("/ffmpeg/trunk/libavcodec/xsubdec.c (http://svn.mplayerhq.hu/ffmpeg/trunk/libavcodec/xsubdec.c?sortby=date&view=log)"), which makes for the first open source XSUB decoder (.divx embedded subtitles) as per this feature request.
http://xbmc.org/forum/showthread.php?t=18071DivX Subtitles (XSUB) (http://en.wikipedia.org/wiki/DivX#DivX_Subtitles_.28XSUB.29)

DivX, Inc. (http://en.wikipedia.org/wiki/DivX%2C_Inc.) (formerly DivXNetworks, Inc.) have since DivX 6.0 their own proprietary (http://en.wikipedia.org/wiki/Proprietary) subtitle tracks that they call "XSUB" (which they also trademarked as XSUB). These subtitles are not text-based (http://en.wikipedia.org/wiki/Plain-text) like many other subtitles, instead they are bitmap (http://en.wikipedia.org/wiki/Bitmap) (digital image (http://en.wikipedia.org/wiki/Digital_image)) based like VSFilter (http://en.wikipedia.org/wiki/VSFilter) subtitles for DVD-Video are. And like VSFilter subtitles for DVD-Video are suppose to, XSUB does not come in standalone files but are only embedded in .divx containers, which can be created with Dr.DivX, (Dr.DivX can actually convert/encode XSUB from VSFilter subtitles inside DVD-Video). A .divx (http://en.wikipedia.org/wiki/.divx) container can contain multiple XSUB subtitles in several languages.

Also, FFmpeg have updated their dvdsub_parser.c (http://svn.mplayerhq.hu/ffmpeg/trunk/libavcodec/dvdsub_parser.c?sortby=date&view=log)/dvdsubdec.c (http://svn.mplayerhq.hu/ffmpeg/trunk/libavcodec/dvdsubdec.c?sortby=date&view=log) (DVD subtitle parser) to be able to parse HD DVD (http://en.wikipedia.org/wiki/HD_DVD) subtitle packets.

[keito]

Has anyone else experienced problems with subtitles not showing at all in the GNU/Linux port? I've tried to use the Arial.tff that comes with XBMC, arial-iso-8859-8, and also DejaVuSans.tta from my OS' font folder; neither would show up. I tried to change charset from Default to Western Europe, but that do anything. I've also checked for each file that the subtitles were indeed activated.

The files I've tried to use are mostly MKVs with softsubs, is support for this not implented yet? I don't get why, because handling those should be mplayer's task, right? Never had any problems on my Xbox hardware.

[Gamester17]

MPlayer has not yet been ported to the XBMC Linux port (nor to XBMC for Mac OS X, nor to XBMC for Windows), XBMC for Linux instead uses the DVDPlayer which does not yet support external subtitles (but only embedded DVD-Video subtitles in DVD-Video, so not even such subtitles embedded in AVI, MKV, OGM, MP4, etc.)

See these specific XBMC wiki articles for more information and updates:
http://xboxmediacenter.com/wiki/?title=XBMC_for_Linux_port_project
http://xboxmediacenter.com/wiki/?title=XBMC_for_Mac_OS_X_port_project
http://xboxmediacenter.com/wiki/?title=XBMC_for_Windows_port_project

[vulkanr]

actually "srt" external subtitles are now supported in dvdplayer thanks to d4rk and elupus...

[Gamester17]

Cool, but embedded text-based (srt, etc.) inside AVI, MKV, OGM and so on are not supported yet are they?

[elupus]

nope, only external at the moment.

[A600]

Will DVD-Video playback support external subtitles?

[embrion]

Hello,

Why only SRT external subs are supported?
All of them work with Xbox version and SRT works with everything so from coding point of view we got both "choose subs file" and "parse subs file" functions already done.

[spiff]

it isn't supported because nobody has stepped up and added it. those functions are part of mplayer's (imo obfuscated) code base. you want it? patch is certainly welcome

[Gamester17]

http://xbmc.org/wiki/?title=Google_Summer_of_Code_2008#Team-XBMC_proposal_ideas_up_for_discussion* Unified Subtitle Parser / Display Filter Library for XBMC's in-house DVDPlayer video-player core
** Create a Unified Subtitle Parser Library for external and embedded subtitles, plus a FFmpeg libavfilter filter (http://wiki.multimedia.cx/index.php?title=Libavfilter) for it Maybe a GSoC student will pick it up, ...one can always hope

[Gamester17]

FYI; here is a related suggestion for XBMC's DVDPlayer => http://xbmc.org/forum/showthread.php?t=6982

That is a request for Advanced SSA/ASS subtitle support in XBMC

[Gamester17]

If any programmer(s) our there is interested in this, maybe it would be worth looking to the idea of implement such a 'Unified Subtitle Parser/Library' as a filter (or several filters) for FFmpeg's new libavfilter API (http://wiki.multimedia.cx/index.php?title=Libavfilter)?, and as such implement it into FFmpeg (http://www.ffmpeg.org) directly, ...and thus XBMC would benefit from this just as all other open source video-player that uses FFmpeg could.

???

http://wiki.multimedia.cx/index.php?title=Libavfilter

This because XBMC's own in-house developed video-player (currently called "DVDPlayer") (http://xbmc.org/wiki/?title=DVDPlayer) uses FFmpeg and has cross-platform support:
http://xbmc.org/wiki/?title=DVDPlayer


http://xbmc.org/wiki/?title=Google_Summer_of_Code_2008#Team-XBMC_proposal_ideas_up_for_discussion* Unified Subtitle Parser / Display Filter Library for XBMC's in-house DVDPlayer video-player core
** Create a Unified Subtitle Parser Library for external and embedded subtitles, plus a FFmpeg libavfilter filter (http://wiki.multimedia.cx/index.php?title=Libavfilter) for it Maybe a GSoC student will pick it up, ...one can always hope :cool:

[bmfrosty]

I've had a couple of thoughts on this.

In all cases XBMC should be able to pass the subtitle file(s) to the parser/render which will determine what format they're in and return an array of timecodes (frame numbers?) and a unique identifier for each subtitle. XBMC should allocate memory for each subtitle a few frames before it is needed, and send a request to the parser/renderer. The parser/renderer would do it's work, and when done would set a flag so that XBMC knows that the subtitle is ready and can be overlayed on top of the appropriate frames. XBMC would determine when the subtitle would no longer be needed and free the memory itself.

All of the text based subtitles would eventually be rendered by libass. Most of the text based subtitle formats have a very limited feature set, and would be converted to ssa/ass format at run time. Libass would then render the the subtitles in memory. If I understand libass correctly, it renders subtitles at the display resolution, so no further scaling should be needed.

Image based subtitles would be handled in a similar way, but with an additional step. After the subtitle type is identified, the subpicture image would be rendered to memory, and instead of being passed directly back to XBMC, it would scaled to the proper resolution first. I believe that mplayer has a filters of this sort already implemented to reduce aliasing on DVD subtitles and make the subtitles prettier overall. Maybe code could be borrowed from there.

Additional thoughts:
Some media formats - MKV comes to mind - have the ability to include embedded fonts. XBMC would need to have a way to either pass the fonts to the parser, or optionally save them to a fonts directory (~/.xbmc/userdata/fonts?) for later use. In either case, XBMC will need to be able to tell the parser/renderer (which will then tell freetype and fontconfig) that new fonts are available, and where to look for them. Another thought is that freetype and fontconfig probably keep some sort of font font database, and if this isn't stored somewhere persistent, it will have to be recreated either every time XBMC is started, or every time a video with subtitles is played.

The parser should be able to deal with multiple subtitle files at any given time - for instance separate subtitles for Karaoke and Dialogue. Additionally the parser should be able to switch subtitle streams on the fly and reload subtitle files that are modified during playback (for streaming and subtitle creation purposes).

The DVD standard allows up to 32 subtitle streams to be present in one stream, and the most intensive use of a subpicture stream that I'm aware of is on the original pressing of ghostbusters. One of the subtitle tracks had a subpicture for every frame of the film for a total of about 151,000 subpictures. The unique identifier should probably be a 32 bit integer with either 6 or 8 bits to identify the subtitle stream, and 24 or 26 bits to identify the individual subtitle (line of dialogue or subpicture).

---

Alternatively, the array of timecodes could be kept entirely within the parser/renderer. In this case, XBMC would have to keep the parser/renderer constantly updated with the current timecode (or frame number) and the parser/renderer would keep XBMC updated with a current list of available subpictures, and their associated timecodes (or frames) and addresses in memory. In this case the parser/renderer would be in charge of allocating and freeing memory for the rendered subtitles and determining ahead of time what subtitles need to be rendered when.

---

There's a lot more detail that could be put into this, and I'm aware that the a lot more information will have to be passed between the parser/renderer and XBMC, but I've been thinking about this for a while now, and here seemed to be as good of a place as any to put it down for the world to see.

[bmfrosty]

Alternative 2 - XBMC would keep the parser/renderer constantly updated with the current timecode (frame) and pointers to the current frame in memory. If the parser/renderer has a subtitle available for that frame, it could set a lock, overlay (or render) the subtitle on top of that frame, and then release the lock. Then XBMC would display it to screen once the lock was released.

How does XBMC currently deal with overlay of subtitles (and other overlays)? Is it handled by the pixel shader?