View Full Version : Should XBMC use system libraries when possible?
This is in response to http://xbmc.org/trac/ticket/5416 since that ticket is becoming way too long and more of a religious discussion then a bug/feature ticket.
@candrews: I didn't mean to scare you off. There's no reason you can't learn and contribute.
@Xake: How is telling people they are on their own if they don't use specific versions any different than providing them with specific versions. And if Ubuntu upgrades their version, does that mean we no longer support the simple users who don't know how to install an older version? And how is it any better to force users to downgrade their libraries? Then they'll break other packages. The last thing we're going to do is give users a compile time option of using their system libraries or ours. Even if we say we don't support it, people will still come here asking for us to support it. If we're going to go that route we're going to go at it full force. And I do think it's a good thing to work toward, we just have to be careful how we do it.
Gimp may not have the full version of the source in their repo, but we're not Gimp so we're allowed to do it different. There is no harm in including the source in our repo. It especially helps with debugging. You don't know how many times I've had to go into Python to track down a crash. A feat which would not have been nearly as easy if we used the system library.
We're not even going to get into security. If you surf the Internet, you're done already. Perhaps we should take out all support for remote media since that has the highest probability of being malicious. Frankly I'd be more worried about buffer overruns in XBMC then in third party libraries.
I guarantee the changes we've made to Python cannot be pushed upstream. If you'd like to write the Python dev list and ask them to please not use certain functions, then by all means. But as I said to candrews, make sure you know what you're talking about before you go making these grand statements. The source is available, do a diff.
@Xake -> "And sorry for maybe being a bit rude, but I find at least davilla's sense for security a bit bad. ".
It's easy to trivialize network security by a blanket statement about needing security updates. But all the security update in the world will not protect from a zero day exploit.
I only expose hardened boxes to the raw internet. I only stream/download from trusted sources. Anything else is unwise and sooner or later you (not me) will risk an exploit. 20+ years of dealing with computers and I've had one successful intrusion.
I for one, am not interested in playing the never ending catch game of security updates. If there's a serious problem, it will be addressed otherwise, the source code is in svn and you can apply any security patches yourself and build your own binaries.
To the other comments -> you are trivializing the build/debug/release process for a large multi-platform source code project. Any changes must consider the impact on osx ,linux, and windows platforms. Instead of just reading the readme for linux, also add the readme's for osx and windows before you start with criticism. And try to understand why certain libs are built and others are ref'd. Lumping them all together is not understanding their dependencies.
Stability is job #1 and part of that stability is controlling dependencies to a certain degree and that is done by using specific versions of some external libs. If you don't like this, then check out from svn and build your own binaries just the way you want them build.
candrews
2008-12-10, 04:43
I'd love to help, cause I realize in this world, it's better to do than to whine. Perhaps someone can point me in the right directions to start doing the debundling work on a simple library, then I could take it from there? XBMC is just so huge, and complex (it's really impressive!), and the build system is likewise complex...
I think XBMC has a better chance of being included in distros this way, and that's really what I'd like to see. :-)
I'd say start with cximage. Figure out if cximage is in Ubuntu. If not, many of the graphics libraries it links against are and should have a fairly stable interface. We already link against a few. See if we can link against them dynamically (by doing it and testing out viewing some pictures). This little exercise will hopefully get you acquainted with our system, and debundle a package in the process.
wattazoum
2008-12-30, 20:10
Please look at this document to know how to wrap at runtime. http://lattice.umiacs.umd.edu/files/functions_tr.pdf .
This could solve the wrapping issue under linux.
Gamester17
2008-12-30, 21:42
All I got to say on the topic is this, please always keep in mind that XBMC is a multi-platform software so remember that all code should strive to be platform agnostic if and when at all possible.
I am not saying that platform specific code or features/functions are necessary bad but I think it makes it easier if XBMC is not too much dependent on platform specific library's, its probably more work but still better do have control over everything that XBMC depends on and therefor package that with XBMC.
My 2 cents
althekiller
2008-12-30, 22:26
Linux is the only platform where this really causes a problem. Windows doesn't have any of the libs we load by default. OSX libs are controlled by macports/fink. Linux has a billion distros with a billion different versions/patchlevels of libs.
Linux has a billion distros with a billion different versions/patchlevels of libs.
But those distros would probably not even think about packing/supporting xbmc in its current state with a lot of bundled libraries.
What Diego from Gentoo (and upstream xine-lib dev) thinks about it:
http://blog.flameeyes.eu/2009/01/02/bundling-libraries-for-despair-and-insecurity
wattazoum
2009-01-05, 20:02
Very good article !
althekiller
2009-01-05, 20:50
But those distros would probably not even think about packing/supporting xbmc in its current state with a lot of bundled libraries.
What Diego from Gentoo (and upstream xine-lib dev) thinks about it:
http://blog.flameeyes.eu/2009/01/02/bundling-libraries-for-despair-and-insecurity
What I said was in support of the changes. They aren't as pressing for win32 and osx because the libs are more tightly managed.
monkeyman
2009-01-06, 05:47
Would it be possible to have XBMC use the system libraries if available, but fall back to bundled libraries where needed?
It would make maintenance easier on the XBMC side if our Linux libraries could be included entirely in the distro.
I think that might be hard but Im unsure, recently I tried to move and link with libsmbclient-dev instead of our bundled version (like we did prior to hardy IIRC)
This is apt-getable and compiles but fails during linking as libsmbclient.a is busted in intrepid, this is a bad example but if not even ubuntu can keep its libraries working throughout the versions then its hard for us to use them :blush:
Would it be possible to have XBMC use the system libraries if available, but fall back to bundled libraries where needed?
It would make maintenance easier on the XBMC side if our Linux libraries could be included entirely in the distro.
Falling back is just as bad as including them in the first place. If something is busted or not included, we should know at compile time.
monkeyman
2009-01-06, 21:04
Falling back was a poor choice of words on my part. I meant: should we include those libraries in builds for OSX and Windows since those libraries may not be available on those systems. However, for Linux, we could use the system libraries?
This probably is not very feasible since we're trying to build a cross-platform media center and consistency between our libraries is going to be critical.
Consistency was the argument for including the libraries in the first place, but most of the libraries we include are stable enough to use the distro version in Linux. Windows and OSX are a different story since there's we can't reliably expect these libraries to be included.
Windows and OSX are a different story since there's we can't reliably expect these libraries to be included.
I think it would be great if you did not have to have all those sources in the tree.
Refering to a specific version or csm-revision, having a place you can drop the upstream source for this package and having the possibility to have xbmcs buildsystem build those libs. Or have the buildsystem with a special switch fetching them also from a known location. That would fix it for those who use platforms without the necessary deps present.
It would make it more of a challange to build the sourcetree for those having to fetch sources from many places, but it would greatly reduce your bandwith I believe if you do not host those sources. Counting those that will use system-libs and not really need those sources would reduce it futher.
jbennett
2009-01-24, 05:57
I think the most significant argument to allowing shared libraries is to get XBMC into various distro's. Without that, I don't think xbmc will get a large linux following.
Would it be useful to start by re-organizing the xbmc source tree so that the XBMC core code and libraries are seperate? We can have a source release of both the shared libraries, and the core xbmc. But they would be packaged seperatly. Obviously, for now anything that is modified would have to stay in the core tree, until we change XBMC or the changes get pushed upstream.
At the very least, is it worth starting a branch of the linuxport tree to add code to conditionally allow using the shared versions of the library? I would be interested in submitting code to this and testing it on windows and linux (don't have an original XBOX anymore).
jmarshall
2009-01-24, 07:20
There's been work done on this by ceros, and a branch already exists in SVN, that includes ffmpeg and I think libcdio maybe using the system libs.
Moving to runtime wrapping is really what is holding this up for a lot of libraries. I'm sure he'd welcome any help.
Welcome to the forums!
Cheers,
Jonathan
The use of external system libraries breaks into two parts:
1) static libs that xbmc.bin links.
2) dynamic libs that xbmc.bin loads/unloads during runtime.
The work being done by ceros addresses 1). 2) requires runtime wrapping to support the xbmc virtual file system.
Runtime wrapping is being worked on; there is a method that can be used under Linux and I'm investigating methods for OSX. I think Windows already does runtime wrapping.
One very important thing to remember about runtime wrapping is that the API that XBMC uses must match that of the wrapped library or bad things happen. This means library version dependencies will need to be taken into consideration. The current method of building/wapping is immune to this as XBMC builds the required libraries from source code using specific versions.
davilla, I found a possible way to do it on OSX a while ago. If you link me to the paper I can find it again.
davilla, I found a possible way to do it on OSX a while ago. If you link me to the paper I can find it again.
This one? -> http://lattice.umiacs.umd.edu/files/functions_tr.pdf
Here we are: http://koichitamura.blogspot.com/2008/11/hooking-library-calls-on-mac.html.
Not the same one I have before, but this one is more awesome because it shows us how to do C++ also! Now maybe we can override iostream.open.
A few more links I found. Mostly linking so they don't get lost, I'm not sure they're useful beyond what the first has:
http://lists.apple.com/archives/unix-porting/2004/Jan/msg00024.html
http://lists.apple.com/archives/Darwin-dev/2007/Nov/msg00062.html
http://www.omnigroup.com/mailman/archive/macosx-dev/2004-December/054995.html
The DYLD_INSERT_LIBRARIES is usefull but is a pain as these have to be exported before running xbmc. That means a helper script that runs the real binary.
I was think more along the lines of mach_override at:
http://rentzsch.com/mach_inject/mach_star12
need to pull the svn version as the web link is not i386.
mach_override will do the job but it's licensed <http://creativecommons.org/licenses/by/2.0/>
so would need to talk with the author.
Would it not be possible to have in the .app a default executable which sets the environment variables and then spawns xbmc.bin?
Don't we do something similar on Linux where xbmc is just a script that launches xbmc.bin. Much like with Firefox. We're going to need to do that on Linux anyway since we'll need to set some environment variables there as well.
possible, I'd have to think about the ramifications not that I can see any right now.
jeremyhu
2009-02-15, 12:45
Linux is the only platform where this really causes a problem. Windows doesn't have any of the libs we load by default. OSX libs are controlled by macports/fink. Linux has a billion distros with a billion different versions/patchlevels of libs.
And how exactly do you come to the conclusion that Linux is affected but not OSX because "OSX libs are controlled by macports/fink" ? If anything, that is a similarity to linux distros and shows how OSX is affected as well...
macports/fink is not part of the base install. we can't count on macports being on an osx install as we can expect portage to be part of a gentoo install. package management is built into every linux distro i can think of.
jeremyhu
2009-02-15, 21:50
possible, I'd have to think about the ramifications not that I can see any right now.
That's actually quite useful. There aren't really any major ramifications. It's what we do for X11.app:
~ $ cat /Applications/Utilities/X11.app/Contents/MacOS/X11
#!/bin/bash
set "$(dirname "$0")"/X11.bin "${@}"
if [ -x ~/.x11run ]; then
exec ~/.x11run "${@}"
fi
case $(basename "${SHELL}") in
bash) exec -l "${SHELL}" --login -c 'exec "${@}"' - "${@}" ;;
ksh|sh|zsh) exec -l "${SHELL}" -c 'exec "${@}"' - "${@}" ;;
csh|tcsh) exec -l "${SHELL}" -c 'exec $argv:q' "${@}" ;;
es|rc) exec -l "${SHELL}" -l -c 'exec $*' "${@}" ;;
*) exec "${@}" ;;
esac
that paper talks about LD_PRELOAD and --wrap. i'm guessing this means that xbmc only works as a dynamic application and will never work statically. i'm ok with that ;).
there are better ways of wrapping functions under Linux though as the ELF spec was designed with proper symbol resolution in mind. that would avoid the hard requirement of a wrapper script setting LD_PRELOAD to a specific library.
how is the wrapping done now on Linux ? modifying these bundled packages and statically linking them in ? is there some documentation on this that i'm missing ?
The purists would like us to dynamically link against the system libraries instead of compiling our own as we do today. Currently we dynamically link against our own compiled versions which have been compiled in such a way that they will use our wrapper functions instead of calling system functions directly. What we'd like is to use the system libraries while still being able to have them call our wrapper functions.
sure, i understand why us Linux lamers want to convert xbmc and why the xbmc guys are doing what they're doing. it isnt like you're doing it because you're lazy :).
i thought i'd start with something simple like curl, but that appears to not be wrapped. libid3tag seems to be modified for other reasons.
so i'm guessing these are source-level modifications to achieve the wrapped behavior ? something akin to s/open/xbmc_open/ ?
based on the thread it looks like this is largely filesystem functions that are being wrapped. i dont suppose you could highlight the dirs in the source tree where these wrapped functions are ?
it's more like compile time modifications. we generally don't touch the source unless we need to, and even then there's probably a better way that hasn't shown itself yet. i believe the wrapping functions are in xbmc/core/DllLoader/export
jmarshall
2009-02-24, 00:12
The main ones on linux are the filesystem ones, yes. Look at xbmc/cores/DllLoader/exports/wrapper.c, and there's another similar (yet different) one specific to python (xbmc/lib/libPython/linux/wrapper_python.c)
On win32 we need to wrap other ones, and on xbox we need to wrap pretty much everything (no dll loader there) so don't get overwhelmed at the shear quantity of crap in DllLoader/export ;)
Cheers,
Jonathan