Porting UnRar library to Xbox/XBMC

[damien7579]

hello developers,

this is my first post, tho i have been lurking around the scene for a wee while - so hello everyone :) i am a big xbox/xbmc fan and user.

i am in the process of porting the latest unrar source code (see http://www.rarlab.com/rar/unrarsrc-3.3.6.tar.gz). the license seems quite relaxed and couldnt see a problem with porting it as a lib...

my main goal is to add support for rar/zip browsing in xbmc with the ability to auto extract a file when selected to a local folder on the xbox hdd which will then be opened as per normal. the unrar lib can also be used to extract subs which is a current todo feature.

before i get to into this, is anyone else already working on something like this and if not and it proves to be a useful addition, how should i submit my efforts in the xbmc cvs tree?

feedback, ideas/advice and so on would be greatly appreciated.

/damien

[thebeast]

try messaging, for one, butcher. also see this thread http://www.xboxmediaplayer.de/cgi-bin....29;st=0 (http://www.xboxmediaplayer.de/cgi-bin/forums/ikonboard.pl?act=st;f=1;t=1429;st=0)

[Gamester17]

feedback, ideas/advice and so on would be greatly appreciated.
i think the major hurdle to overcome will be on-the-fly parsing so xbmc don't have to extract the whole file(s) to harddrive or memory first to read the uncompressed file, but instead being able to extract parcial data from exampel rared vobsubs or other archives. i'm not a programmer myself so i don't know how that is possible to do but complex tools does it and we got this source code (link) (http://www.xboxmediaplayer.de/downloads/temp/cpx-xbox-unrar-sources.rar) from them, noone on the xbmc team has yet had time to look at it so be great it you give it a go.

[damien7579]

thanks for the replies thebeast and gamester17. i will look at the complex rar code and see what can be done.
in respect to partial extraction - it is possible to extract from a rar to memory but depending on the rar settings, some files
will require all the compressed data before them to be decompressed so that the stream can be read - so even if this is all done in temp memory it could be slow - but as long as the files are small this should not be a problem - which is fine for most rar'd subs.

regards,

/damien

[damien7579]

the complex unrar source is based on an oldish distro of the rarlabs unrar source code - but the changes required to port to the xbox are pretty small so i have continued working on the current unrar release. since this is my first attempt at xbox development i have quickly come to realise how important it is to be able to do real time debugging :) - so tonights exercise is to convert my retail box to a debug dev one before continuing. so wish me luck... muhahahahahaaaaaaaa.



/damien

[Frodo]

sounds good
you're gonna make ppl happy with this:-)
frodo

[damien7579]

frodo: yeah i noticed all the posting's for rar support.

in repsect to rar browsing, how difficult would it be to to add support to the smb/local directory classes to allow a rar file to be navigated ? i have yet to look to deep into that but i cant seeing it being a major issue...

anyhoo, thanks for the support :)

/damien

[erpi99]

can you make the unrar also avaible for extracting files into xbox hard drive? it will be very convenient for the upgrade of xbmc. it takes much less time to ftp one tar file into xbox and untar the file in the xbox then ftp thousands of files untared in pc to the xbox.

[damien7579]

the unrar source only supports rars (obviously :) but is there also need for zip support or even other formats?

[Gamester17]

is there also need for zip support or even other formats?
i think butcher said he would look into porting zlib (http://www.xboxmediaplayer.de/cgi-bin/forums/ikonboard.pl?act=st;f=4;t=1810) (for unzipping) to xbmc, but not sure how fully he plans to implement it

[Butcher]

will see once the rar stuff is in what is required. i'll make it at least similar in capabilities to rars (in whatever format we get them).

nothing for a day or two though till i recover from the cold i have.

[thebeast]

the unrar source only supports rars (obviously :) but is there also need for zip support or even other formats?
of course there is, and also lha, for those old amiga mod archives...

[poing]

good luck with this damien, i would love to see unrar-on-the-fly support, a lot of movies/tv episodes are stored in 15-20mb sized rar volumes. m0 compression is almost always used so it's just plain data with a rar header, so there should be little cpu overhead. it would be great to point at the first .rar of a movie and have xbmc start playing it (even if seeking may be problematic).

i could imagine two modes - on-the-fly playing of media from rars and extraction of files to the xbox hd in my files, i.e. entering .rar files like a directory and then being able to copy?

[damien7579]

hi poing.

what you mentioned is pretty much what i was looking at doing. the seeking issue will be a problem - but perhaps for uncompressed rars this can be a special case.
i was basically thinking of the following unrar 'features'

1 - extract small files straight to memory (for sub support)
2 - rar file virtual browsing
3 - extract single file to temp folder on xbox for playing
4 - on-the-fly rar extraction if possible (prob only on uncompressed files due to seeking issues)
5 - extract entire rar to specific folder on xbox

out of interest, do most of the video codecs require seek operations to the end of the file before decoding starts?

feedback welcomed :)

currently the unrar lib has ported fine and am presently working on adding suitable wrapper classes.

/damien

[Butcher]

on the temp space - we have 768mb of temp space to play with (drive z:) so this would be suitable for a lot of cases.

[Gamester17]

1 - extract small files straight to memory (for sub support)note that "on-the-fly rar extraction" of compressed rars is needed for vobsubs too as most are larger than the free rar memory (see here (http://sourceforge.net/tracker/?group_id=87054&atid=581841&func=detail&aid=817655))

out of interest, do most of the video codecs require seek operations to the end of the file before decoding starts?[16:25] <gamestr17> maybe you can answer "out of interest, do most of the video codecs require seek operations to the end of the file before decoding starts?"
[16:25] <_frodo_> yes
[16:26] <_frodo_> avi for example got the index @ the end of the file
[16:26] <_frodo_> thats read first
[16:26] <_frodo_> and cached in memory
[16:26] <_frodo_> so you'll when starting an avi you'll see seek operations
[16:26] <_frodo_> to beginning of file to read all headers
[16:26] <_frodo_> and to end of file to read the index
[16:27] <_frodo_> mpeg doesnt contain any index,
[16:27] <_frodo_> just 1 seek to determine end of file & filesize
[16:27] <_frodo_> so yes, most codecs need seek operations
[16:28] <_frodo_> or
[16:28] <_frodo_> you can disable seeking
[16:28] <_frodo_> but then you cant do ff/rw
[16:28] <_frodo_> or jump to a specific time in the file
[16:28] <_frodo_> since the index is unknown

[thebeast]

out of interest, do most of the video codecs require seek operations to the end of the file before decoding starts?
if most is: avi, asf, mov i think that's a yes.

mpeg-2 seems to play sometimes using only incomplete files.

.rm sometimes play even if there's no end part.

videolan plays a lot of incomplete and broken files, if you are curious to look at other players ;)

[poing]

damien, mpeg1/2 including dvd vob files and complete dvd image files don't require anything from the end of the file in order to seek. divx/xvid however have an index at the end which is required for seeking. mplayer has no problem playing incomplete divx/xvid files, seeking is automatically disabled in this case (there is an -idx option to generate an index from what's available, but that's probably not useful here. i'm not sure if xbmc currently plays incomplete .avis like mplayer). playing divx from m0 rars could perhaps be done without proper rar support, i.e. the last 20mb of data will definitely be in the last rar volume. i'm not sure how the rar format works, but perhaps the header of each volume contains information about the current offset in the uncompressed file? or perhaps the end of the rar header could be easily detected somehow and the rest of each volume can be assumed to be plain data, and then it would just be a matter of counting and adding.

even without seeking, i think the ability to play avi/mpg/bin(mpeg1/2) videos directly out of rar volumes would be greatly appreciated by a lot of people.

[damien7579]

thanks guys for your useful feedack - lots of fuel for thought :)

even without seeking, i think the ability to play avi/mpg/bin(mpeg1/2) videos directly out of rar volumes would be greatly appreciated by a lot of people.

agreed :)

[Xusen]

there is an xbox dashboard with rar support so maybe you could ask the developer there for some assistance where you encounter problems.

i am talking about unleashx dashboard.

[silentyl]

so..any1 made progress on unrar support?

would be great to have downloaded .rar files using the bittorrent script(within xbmc) and extract any .rar to the xbox hd without need a of a pc.
..now the linkbok's www browser is working i can find the "source" line and put in a "destination" line, then use a program like xpad(like notepad) to edit the bittorrent.py file with these details

...i see big potential for unrar support

bittorrents script is found here :
http://www.gueux.net/xbmc-scripts/index.html

by the way ..thanks hulla! :)

[damien7579]

been hell busy at work so havent had time to post progress till now. basically the unrar lib has ported ok except for one last problem which has had me stumped for ages - stack exceptions :) .... unraring files works flawlessly on the xbox using the lib, but if the rar archive list function was called - wam... exception central... it wasnt until i had some more free time and looked at other xbox projects did i find the useful /stack link option which solved the problem. *next step - xbmc wrapper classes. stay tuned!

/damien

[silentyl]

great news dude!
will be a huge step forward for the xbmc...cant wait to see it in action
keep up the good work :)

[thebeast]

looking forward to unrar support!

[Striker]

any progress? http://www.xboxmediaplayer.de/forums/non-cgi/emoticons/wink.gif

[poing]

i made a hack that works together with the ccxstream (xbmsp) linux stream server, it works fine for playing video straight from .rars but has some limitations. proper built-in rar support would still be nice. see http://www.xboxmediaplayer.de/cgi-bin....;t=2685 (http://www.xboxmediaplayer.de/cgi-bin/forums/ikonboard.pl?act=st;f=5;t=2685)

[tric]

short addon to poings port: for xbms users theres a patch too (if anyone still uses xbms and im not the only one ;))

[Gamester17]

fyi, unleashx has ported unrar utility 3.3.6, see this x-s forum thread (http://forums.xbox-scene.com/index.php?showtopic=200415) plus been fixed by benjeremy (source code download mirror (http://www.xboxmediacenter.com/downloads/source/unrarxlib_by_bj-15-04-2004.rar))

[Gamester17]

@devs; fyi, looks like mplayer (http://www.mplayerhq.hu) added unrarlib (http://www.unrarlib.org/) to thier cvs, i'm not sure for what reason (or why they choose unrarlib, it's quite old).
i think/guess they may have added it for the sole reason to read ( uncompress / unpack) vobsub's directly from rar archive, or?

[Guybrush]

yes mplayer uses unrarlib to read vobs directly from the rar, but doesnt always work on some occasions (unknown compression type).

[Gamester17]

yes mplayer uses unrarlib to read vobs directly from the rar, but doesnt always work on some occasions (unknown compression type).looking at www.unrarlib.org (http://www.unrarlib.org/) it cleary state that unrarlib only support winrar 2.x compressed files (so not winrar 3.0 or later) = it's old.
in any case it does not look like damien7579 going to finish his port(?) or even post code-to-date(?) so maybe someone can take over?

[swmike]

what i personally would like to see is all this built into the ccxstream server (or equivalent). ie, do all the hard work at the server side and make it all transparent to xbmc.

but as someone else earlier in the thread noticed, rarlib doesn't support v3 files which makes it a hassle. i guess one could make a unrar frontend builinto ccxstream, but i have a feeling it wouldn't be very pretty, but i guess it would work.