Developing MeD's Movie Manager interface to XBMC - assistance is needed

[phunkyfish]

Hi,

I'd like to start developing an interface between MeD's Movie Manager (pulls movie info from IMDB and stores it in a MySQL database: http://xmm.sourceforge.net/index.php?menu=main) and the XBMC. Currently my main concern with XBMC is that the movie database is stored on the XBox. I would like to have a central database using MeD's that a number of XBox's could query. Each entry would be able to return the file path, thereby allowing you play movies, but more importantly, it would allow to search through your movies for what you want, search by actor, genre etc. whatever you like. You would also have the option of viewing other information about the movie. Sifting through a file browser for the movie you want has now become a very large task...

I was going to start with a Java Servlet with which XBMC could communicate. But I need to know where to start. What is the best way for XBMC to perform queries? Would a list of movies returned in XML format be appropriate? I'm not really concerned about the XBMC end at the moment, I'll focus primarily on servlet side of things to start. I'll be able to get a start on it and leave figuring out XBMC development till a later date.

I'm a Java programmer with experience with servlets primarily. I have worked with C++ in the past.

The beauty of this method is that MeD's is also Java based, thereby allowing it to work with any future ports of XBMC!

Any help would be great, thanks!

[jmarshall]

Hi there, and welcome.

First off, one presumes you've actually tried XBMC's library features out. We use sqlite to do the db side of things, which in it's current form cannot connect to a non-local database (to my knowledge - if you know better, please let me know!). Converting sqlite stuff so that it runs as a dll will open up the possibility of having a non-local database, so that's one way you may want to look at.

Otherwise, something like XML would be a suitable method of passing stuff across. With the nice new virtual python "plugin" folder system we have under way, this would allow you to integrate it with XBMC without having to code any C++ at all (i.e. all done in python) and I would suggest that is the way you go, as I'd think that we'd be not all that keen on having it in SVN for a feature that will be useful for only a small amount of users.

Cheers,
Jonathan

[phunkyfish]

Excellent! That's exactly the answer I was looking for! I'll be back once I have a suitable servlet ready.

I think using XML would be the best idea as it would allow communication with other software not just XBMC. I'd like the info server to be as portable as possible.

Almost all the people I know who use XBMC view there files using a network share on a PC. This would pretty much allow anyone to have an info server for there media. Could easily be expanded to include TV shows also.

Guess I'll be learning some Python in the near future!

Thanks for the quick reply.

[blaize]

wouldnt it be usefull if we could load multiple databases then ?
like when someone connects to several computers, and all those computers have their own database, xbmc all loading them into one.

[phunkyfish]

Your already seeing where I'm going with this!!!

This is why I think XML would be best. Once the schema of the XML is formalised, and generified, you would be able to easily combine the queries. Then, no matter where the data comes from, whether it already is in XML, from a database or even text files you would have everything. Plus it would be database independent.

It will also let each media location be responsible for maintaining it's own media and the information about it. A simple update could be run on all media locations, updating the database in XBMC. Then all searches could be performed locally.

In fact it would be quite easy to write a generic servlet to access data of any type by using a data persistence layer. By simply throwing Tomcat onto your media PC and writing a config file, XBMC would be able to comunicate with it.

There's potential there for this to be very cool!

[Gamester17]

my main concern with XBMC is that the movie database is stored on the XBox. I would like to have a central database using MeD's that a number of XBox's could querywhy not write an python-script for the XBMC client-side and a interface/parser on the MeD server-side to sync the two databases on a regular bases (or on demand). So instead of having XBMC querying the MeD database per movie you just syncronuze the whole database and then just let the end-users use XBMC own query functions to query the local database.
http://xboxmediacenter.com/wiki/index.php?title=Building_Python_Scripts

???

Almost all the people I know who use XBMC view there files using a network share on a PC. This would pretty much allow anyone to have an info server for there media. Could easily be expanded to include TV shows also.I do not see how (or why) you would run this via file-mode in XBMC, you would have to use library mode.

Like Jonathan said/asked, have you tried XBMC in library mode?, it already does all you want, only locally, see:
http://xboxmediacenter.com/wiki/index.php?title=The_XBMC_Database
http://xboxmediacenter.com/wiki/index.php?title=Video_Library
http://xboxmediacenter.com/wiki/index.php?title=How_To_use_Scrapers
http://xboxmediacenter.com/wiki/index.php?title=IMDB

[phunkyfish]

why not write an python-script for the XBMC client-side and a interface/parser on the MeD server-side to sync the two databases on a regular bases (or on demand). So instead of having XBMC querying the MeD database per movie you just syncronuze the whole database and then just let the end-users use XBMC own query functions to query the local database.
http://xboxmediacenter.com/wiki/index.php?title=Building_Python_Scripts


Yes, I think this is exactly what I want. And as below you could sync with MeD on multiple PC's if you so wished. I would only be using it on a single file server, but it should allow you to be able to specify multiple MeD/Other info server database locations...


I do not see how (or why) you would run this via file-mode in XBMC, you would have to use library mode.

Like Jonathan said/asked, have you tried XBMC in library mode?, it already does all you want, only locally

Sorry, I did not explain myself fully. I use the file manager currently because the scrapers don't work very well. I have used the library in the past but it is somewhat lacking. The main problem with XBMC is getting the correct data into the library. Once the correct data is in there the Library functionality is almost perfect. Let me explain:

1) Movies - When you scan a bunch of movies into the library a fair portion of them will be scanned incorrectly, i.e. the info for the wrong movie will end up in XBMC, as an example, Apocalypto will come up as Apocalypse Now. Some scrapes are far worse, with no resemblance to the correct movie at all. It's about 50-50 over a hundred movies or so, half right, half completely wrong.
Also there is no way of selecting multiple genres, how would you show all "action comedies" or show all that are action or comedy?

2) TV Show - I have my shows organised in the following directory structure: "TVShowName->SeasonNumber->Files", each file is then labelled "1x11-showName-otherText.avi". If I try to scan in one directory (i.e. one show) all I get is a pile of folders labelled as numbers and nothing else. Selecting the "folder contains one TV show" option makes no difference. I guess the show name should be taken from the initial folder name and season and episode from the avi file. I could rename all the TVShow avi's so they can be processed by regular expressions... But that's a rather large job... Anyway, not really bothered by TV shows. My directory structure works well enough.

I have a few ideas on how to sort the problems with the movies. One is to have a "video.xml" in each directory stating the "movie name", "year", "imdb ID" and "video files", could have more information if you wish. This would make the scraping task much easier as you know exactly which movie it is. The "video.xml" could have details on any scrape site, and could be extended for use with TVShows also. It would make directory structure irrelevant. It's quite easy to add a video.xml to each TVShow/Movie as it's added. But creating them for every Movie/TVShow when you have a lot of them could be a huge task. However, it would yield better results than the current scrape system. Also, if directories are moved it would be easy to relink the file location to the database. Bear in mind this is all without ever connecting to an external info server. If you think this idea would be worthwhile, I'm already writing a schema to incorporate all video files, any input? Note that this is only for simple Movie/TVShow info. It would just enable better scraping.

The reason for the info server is to be able to have a central location for the information. If your XBox dies, the info is gone. Plus if you've watched a movie on one XBox and then on another XBox in the house it's marked as unwatched. If this information is stored centrally it makes more sense. There would even be option of adding multiple "watched/unwatched" users to the info server. So even if you watch stuff on different XBox's it will still show what you havn't watched correctly no matter who you are.

Let me know what you think? I'm going to use the "video.xml" idea with MeD's, if we formalise the schema between XBMC and MeD's all the better, you would have the option of supporting it in the future. You could even have an "only scan video.xml" option?

[phunkyfish]

This is ridiculous. I'm trying to post a reply, if I enter the reply I want it will never display, even if I remove the quotes. Anyone encountered this problem before?

[jmarshall]

Have approved your post - not sure why it's being moderated. Note that if it happens again, there's no need to keep replying - it will be approved as soon as it's noticed. Will ask the admins why it's being moderated and see if we can sort it out.

[phunkyfish]

Scratch that. It appeared after an hour? I hope it doesn't come up multiple times now. Any moderators around feel free to tidy up any erroneous posts. I hate wasting page real estate!:laugh:

[jmarshall]

Some answers:

1. XBMC's database is SQL - thus, any query you like can be constructed as long as the code is added to construct and execute said query. I suggest extending the music Smart Playlists is the way to go in this regard - it allows basically anything.

2. XBMC will support that filesystem layout for scanning TV shows as long as you set up the regexp so that it'll work. You want to assign TV shows to the folder that contains the <tvshowname> folder, or assign "single TV show" on the <tvshowname> folder. The default regexps will then work I suspect with your show filenames. You are just setting content on one folder too far in, so it looks at "1" as the show name.

3. XBMC already supports ".nfo" files which can be XML with all the stuff that IMDb and other sites can return - see the online manual for more, and you can export the db as XML.

4. As far as XBMC goes, I would personally like to see how SQLite responds over the network - having XBMC's own native db file available on the network, assuming the retrieval isn't too slow would be the way to go IMO for multiple installs. As I said, this can be done in one way by building sqlite as a dll, thus the filesystem stuff will be transparent, and sqlite will see it just as a local file no matter where it is. I'm not sure if this can be done in a more efficient way or not, as I'm not a db guy.

Hope this answers some of your questions - I'm interested to see where you see this going - many have requested such a feature in the past, but noone has yet come through with a solution that works gracefully.

Cheers,
Jonathan

[ralob]

good luck with this, i am anxious to see how it turns out and will be watching it closely. :)

[phunkyfish]

Cool, it seems you guys have already done a lot of the work for me!

Firstly SQLite:
-------------

We know that SQLite can't access a remote database. But is the converse also true? Can a remote machine, let's say a PC, access an XBox's local database ? I mean without copying it over, editing and copying back?

Concerning TV Shows:
---------------------

I'd say I just need to create an "advancedsettings.xml" to add a new regular expression for filenames like "1x05-showname.avi". There doesn't seem to be an existing expression for filename starting with the season number they all appear to want some text first. Quick question, are the regular expressions in "advancedsettings.xml" processed before or after the default ones? I'm presuming before... I'll get back to you on this if I don't have any luck.
Since your only searching for Season and Episode numbers why by default expect a "." or a "-" first?

The following expression:

<regexp>[\._ \-]([0-9]+)x([0-9]+)[^\\/]*</regexp> % foo.1x09*

could instead be:

<regexp>[\._ \-]?([0-9]+)x([0-9]+)[^\\/]*</regexp> % foo.1x09*

matching 0 or 1 dots or dashes removing the need to add an "advancedsettings.xml"?

And moving on:
---------------

As far as development goes I think I'll stay away from SQLite as a DLL. It would be the ideal solution in a way but it's been a while since I've used C++ and was always a Borland guy at the time! We didn't like DLL's very much:laugh:

I'm going to try converting the information required to XML, eventually coming up with a generic format for library information. I'll use the NFO method for scraping, so that it can work automatically and correctly. I'll use the same NFO method that XBMC uses with MeD's. Then I will be able to use Python (used it in it's infancy a long time ago!) to add the XML info to the library. I'll make this work both ways, so that both MeD's and XBMC can be converted to the same XML format so that both XBMC->MeD's and MeD's->XBMC are possible. At his point combining multiple XML files into one if there are multiple media locations should be easy.

Finally I'll try expanding the search functionality in the library to allow more complex searches. Not sure if I'll use python or delve into C++ for this. Advanced searching may be something that everyone might want so C++ might be the way to go on this. Either that or I'll bug some of you guys into implementing it!!! But this will definitely be the last thing done.

Anymore more feedback/direction is very welcome...

[jmarshall]

1. Regarding SQLite: If it's built as a dll it'll have no problem accessing a db over the network. This is IMO the first thing to try as far as performance goes. I'm not sure how well it'll work as a DLL but that would be the way I suggest would be "easiest" to do. Alternatively (probably easier), you'll need to build it so that it uses the XBMC filesystem (CFile) instead of whatever it normally uses (probably just standard I/O) - this will mean it includes stuff from XBMC, so will require it be included as a separate project within the solution, but that's no problem.

Getting something to connect to the local (on xbox) SQL db is not really doable, unless you want to completely replace it (i.e. just copy a new version over, or run an Import from XML).

2. Regarding TV show regexps - obviously we're open to altering the defaults if there's a reasonable need to do so - will chat with spiff next time he's around.

3. Via python or C++ is fine for this - the Music Database exposes a general SQL query interface (possibly that only returns fileitems - haven't really looked at it), and I expect the Videodatabase either already does or can be made to do so. Combining this with what we have in the python plugin folder stuff will mean you can do lots of fun stuff.

Cheers,
Jonathan

[phunkyfish]

With the advent of XBMC doing all scraping correctly maybe I could to do this with just a servlet and a python script. The servlet could use the XML file as it's database it would require a little more intelligence on the servlet end, but it's possible. At a latter stage a database backend could be added to the servlet. Or even use SQLite as it's database. Come to think of it, the servlet could just use the XBMC database. When the point came that XBMC's SQLite DLL could access remote databases integration would be flawless. No point in creating a new database layout if I don't need to. At this point the servlet could just be used for remote access of the database so you could view your library when off the local LAN.

Yup, just checked, there's a Java wrapper for SQLite. Heehee, just read the post again, you can see my thought process change as I wrote it!!!

:laugh:

[phunkyfish]

Been looking at SQLite. All I can find is how to cross compile it on Linux?

Why does it have to be in an DLL as opposed to the way it currently is (which is what?).

What kind of setup will I need to compile it? Will I need all of XBMC compilable in order to do this?

[jmarshall]

For Win32, you'll need to compile it with the solution file in SVN (docs/ folder).

You'll either need to compile it as a DLL (possibly won't work, depending on whether it uses stuff like string internally, as our DLL loader doesn't support a lot of template stuff) or change the filesystem layer in it to use XBMC's filesystem (including CFile etc. as necessary) and building as a .lib as it is now.

You may need all of XBMC compilable (for win32 is fine) to do it, yes. For win32, you require the DirectX SDK and visual studio .net 2003. This is probably the preferred method.

For xbox you require the XDK (and VS.net 2003).

Ofcourse, if you are on linux, you can use the linuxport branch and hope that someone kind will port it back to xbox/win32 for you.

As I say, this may or may not be a simple exercise - I haven't really looked into it - I'm just the ideas man here :p

Cheers,
Jonathan

[pike]

phunkyfish, on behalf of our forum let me apologize. we use a solution to catch spammers and when someone is below 10 posts, they are in the dangerzone of getting caught automatically. we also relaxed the rules slightly.

[phunkyfish]

phunkyfish, on behalf of our forum let me apologize. we use a solution to catch spammers and when someone is below 10 posts, they are in the dangerzone of getting caught automatically. we also relaxed the rules slightly.

No worries! I think it's better you catch the spammers!

[phunkyfish]

Ok, I have one more question. You are able to edit your XBMC databases on a PC. But you need the XBMC specific "sqlite.dll"(http://xboxmediacenter.com/wiki/index.php?title=HOW-TO:_Use_your_computer_to_edit_XBMC%27s_%28SQL%29_d atabase-files).
Now, if I use the Java SQLite wrapper with this I should be able to create a servlet which accesses an XBMC SQLite DB. But if you already have an SQLite DLL could we not just use this?

Oh, are there schemas/layout descriptions for XBMC's database's? Or do I just use the DB as a template. It would be nice if there was some way in which I could stay up to date with DB changes so the servlet won't break everytime the DB changes. Or does it change very often?

"Idea man" jobs are the BOMB :rofl:

[jmarshall]

You no longer need the XBMC specific sqlite.dll. Just a regular SQLite v3 dll will do the trick for reading stuff on the PC.

It won't (most likely) be loadable from XBMC's DllLoader, but I don't really know as I've never tried. See some of the (many) dll interface files (eg DllImageLib.h) and see if you can get the sqlite dll to load and do something useful. If you can, then that's definitely the way to go.

I'll fix up the manual.

DB doesn't really change that often as far as layout goes.

[phunkyfish]

Do I have to use version 5849 of the XDK? Or can I use 5933?

Can't get a copy of 5849 anywhere... Hoping the manual hasn't been updated on that front either:grin:

[jmarshall]

5933 is fine.

I highly recommend you testing on the win32 version though - it's a much less painful experience (way quicker to debug when it's on the local machine than over the network).

With that said, however, debugging over the network using the XDK is remarkably good, assuming you aren't using wireless.

[phunkyfish]

Hang on. So I can run XBMC on a PC? I thought that was just an emulator for Python scripts???

I think the learning curve is getting steeper...

I can't find a page on running XBMC on a PC? Is there one or am I getting myself mixed up?

[phunkyfish]

Sorry I meant to say "find a page on building and running XBMC on a PC".

Cheers for all the hand holding!

[jmarshall]

1. Grab SVN.

2. Open Tools/Win32.

3. Edit run_me_first.bat to suit your paths (if necessary).

4. Open the XBMC_PC.sln up and away you go.

You (just) need the DirectX SDK installed - anything from summer 2004 update and above should work fine.

Most things work, and it's what I use for development 99% of the time.

Cheers,
Jonathan

[phunkyfish]

Cool, that makes things a whole lot easier. I'll give a last tomorrow when the SVN checkout is complete... And thanks...

[phunkyfish]

I have this months DirectX SDK. I'll post again tommorow if (when!) I have any problems...

[Gamester17]

FYI; more on XBMC_PC (XBMC for Windows/Win32): http://xbmc.org/forum/showthread.php?t=23235

PS! Feel free to add a article to the XBMC wiki manual on HOW-TO: Compile XBMC for Win32 from source code (http://xboxmediacenter.com/wiki/index.php?title=HOW-TO:_Compile_XBMC_for_Win32_from_source_code&action=edit)

[phunkyfish]

FYI; more on XBMC_PC (XBMC for Windows/Win32): http://xbmc.org/forum/showthread.php?t=23235

PS! Feel free to add a article to the XBMC wiki manual on HOW-TO: Compile XBMC for Win32 from source code (http://xboxmediacenter.com/wiki/index.php?title=HOW-TO:_Compile_XBMC_for_Win32_from_source_code&action=edit)

Will do! It'll be the end of the week before I'll have a copy of Visual Studio. As I have never used Visual Studio I'll write some wiki pages on various tasks I come across, this should give a total NOOB perspective on compiling/writing stuff for XBMC win32...

[phunkyfish]

When I'm looking at the SQLite DLL issue, I might also have a look at abstracting the database functionality of XBMC. I would imagine that XBMC uses fairly standard SQL. It should be possible by introducing a persistence layer to have XBMC perform queries on either an SQLite DB or another Database, such as MySQL which could be located anywhere. If all SQL queries follow the same path: "XBMC query->Persistence Layer->DB". It should be possible to have numerous Persistence Layers for various DB's once the interface to the persistence layer is generic enough.

I havn't looked at any of the code for XBMC yet, nor have I built it yet. Once I get a chance to delve into the code I'll see which option would be best. A persistence layer could be a huge undertaking depending on how the DB side of things is currently written.

[Gamester17]

To bad that XBMC's embedded WebServer (libGoAhead from GoAhead Software) do not support PHP, then you could maybe have used phpMyAdmin (https://sourceforge.net/projects/phpmyadmin) or something similar to access the databases.

As it is the GoAhead WebServer only support Active Server Pages (ASP), Embedded JavaScript, and in-memory CGI processing, ...though it is also possible to use spyce (python server pages), and AJAX without JAVAhttp://xbmc.org/forum/showthread.php?t=1358

Maybe there are some other similar tools to phpMyAdmin (https://sourceforge.net/projects/phpmyadmin) that preforms the same tasks for SQLite but is written in a language supported by GoAhead WebServer?

[RockDawg]

phunkyfish - A little tip I learned for sanning in movies is to create a .nfo file for the given movie. You don't need to enter all the info for the movie. All you have to do is enter the IMDB url for that movie's info and then when you do a scan XBMC will be sure to get the correct movie information everytime. It's really quick and easy.

[RockDawg]

By the way... I will be anxiously watching the proggess of this. I, along with many others I'm sure, would kill to have a remote db for multiple xboxes. Good luck and thanks for taking this on.

[jimg]

I just want to echo RockDawg's statement, I will be anxiously watching as well.

[phunkyfish]

Hi Guys,

Sorry I havn't been on for a while. Works been keeping me busy. But there has been some progress.

I've managed to interface Java to the SQLite DB. So here's the solution I'm working on:

1) Setup each XBox on your LAN in the servlet.
2) Select one as the master. Other's are slaves. The master collects the info from IMDB. The servlet then copies (via ftp) the DB to the PC. The PC then copies the DB to all the slaves.
3) You may browse the entire DB from the servlet (where it is local).

Note: All the ftp'ing etc. will be done by the servlet. No need to manually copy stuff. The idea is that once you have setup the XBox's in the servlet the rest is automatic.

Now, for the questions:

Q1) Will there be a problem copying the DB from one XBox and just putting it on another? I understand I'll have to move related files such as DVD covers etc as well as the DB. But is there a situation where this will not work. Bearing in mind that no media will be stored locally on the XBox. It will all be a on a fileserver.

Q2) Is there an easy way to find the paths to DB files etc. on the XBox remotely (Such as if it's installed as a dashboard or application, or which drive it's on)? Or should I just require the user to input the paths when setting up an XBox on the servlet. It would be nice if they didn't have to.

I originally had started with getting SQLite working over the network. The PC I was working on gave up the ghost and I had already given back the Visual Studio CD's. Plus the guy I got them from was away so I decided to move on with a different solution.

I like this solution though. It can evolve as a separate project and leaves the local DB on the XBox so there will be no DB access time concerns. Plus you can access what's in your collection externally via the web.

What do you think the chances are that you could add a number of user's in XBMC so that watched/unwatched could be setup for multiple people? Could be no users by default. But the facility to add users would be handy I think. Especially when using the servlet to manage multiple XBox's. This would require synchronising the DB's from all XBox's, but it's really a future concern...

Oh, and jmarshall did you talk to spiff about changing the regexp's for TV Shows?


2. Regarding TV show regexps - obviously we're open to altering the defaults if there's a reasonable need to do so - will chat with spiff next time he's around.

[Gamester17]

FYI, I subitted these two ideas as feature requests directly to MeD's Movie Manager:

Use XBMC's XML scrapers for HTTP scraping - RFE:
http://sourceforge.net/forum/message.php?msg_id=4553533
Use XBMC's XML scrapers for HTTP scraping - Forum discussion:
http://sourceforge.net/tracker/index.php?func=detail&aid=1807913&group_id=83511&atid=569698

Export database to individual NFO files - RFE:
http://sourceforge.net/tracker/index.php?func=detail&aid=1807927&group_id=83511&atid=569698
Export database to individual NFO files - Forum discussion:
http://sourceforge.net/forum/message.php?msg_id=4553574

Hopefully some developer(s) will someday like those ideas and implement them :cool:

[Gamester17]

@phunkyfish, have you made any progress on this?

[phunkyfish]

Yes and no, I think I have a pretty neat servlet interface to the SQLite DB now. Unfortunately I've been working out of the country the last few months and it doesn't look like I'll be back to normality for another month or two yet.

But no-one seems to have answered the questions I posted... Bearing in mind I havn't been next to my xbox in 3 months now, it's hard for me to answer them myself...

[Gamester17]

Q1) Will there be a problem copying the DB from one XBox and just putting it on another? I understand I'll have to move related files such as DVD covers etc as well as the DB. But is there a situation where this will not work. Bearing in mind that no media will be stored locally on the XBox. It will all be a on a fileserver.I am not 100% sure but I do not think that will be a problem, you can follow the similar discussions/questions in these two topic threads:
http://xbmc.org/forum/showthread.php?t=15971
http://xbmc.org/forum/showthread.php?t=29299

Q2) Is there an easy way to find the paths to DB files etc. on the XBox remotely (Such as if it's installed as a dashboard or application, or which drive it's on)? Or should I just require the user to input the paths when setting up an XBox on the servlet. It would be nice if they didn't have to.I think that XBMC built-in commands API for python scripts does have a way and what the python API supports the HTTP API should support as well, see: http://xboxmediacenter.com/wiki/index.php?title=WebServerHTTP-API
Maybe ask a question in the python development forum for confirmation on that if you can not find the command in the HTTP API.
...or maybe talk to nad about the HTTPAPI or Nuka1195 about built-in commands in python.

[phunkyfish]

Cool, thanks for the reply gamester ;)

As I said I'm away at the minute. The interface to the DB is quite rich now and I'm thinking of moving it over to Struts (more because I don't have an xbox with me so I won't be making progress with the other stuff!)

Should have something for everyone to play with some time after Xmas. It will be as simple as install a program (any servlet container, tomcat, jboss, etc, not to worry there all free), copy a file and open a browser... (he says with his fingers crossed!).

The only thing is I don't make things that look nice, someone else may of course do there magic on the final version and jazz it up...

[Gamester17]

Do you think that XBMC would benifit from using ADOdb (or better yet ADOdb Lite)? ???
http://en.wikipedia.org/wiki/ADOdb
http://en.wikipedia.org/wiki/ADOdb_Lite

From what I understand that would not only make it simpler for XBMC to change the database backend if needed (if someone for example like to fork XBMC Linux port to use MySQL instead of SQLite), but it should also more importantly enable better direct access for the Web Interface (HTTP API) and Python to the database?

[phunkyfish]

I've written the Java module with persistence in mind, so it should be possible to use any DB, not just SQLite. Bear in mind that SQLite will work nicely because it's just a file you can copy anywhere. Using other DB's you will have to remove/create the DB's each time. Let's get the first version running and tested a little and then we can move onto other stuff. I'm expecting to have a first release at the start of February. I'm sure we'll have a host of teething problems at the time that may take a little while to sort. Still have no idea what effect replacing a DB in XBMC will have when it is running or accessing it's library. We may need a HTTP call/handle for start/stop update or something like that, but once more initial version first...

[Gamester17]

We may need a HTTP call/handle for start/stop update or something like thatGood idea; a "lock_database" and "release_database" mechanism for each database sounds like logical function to have, (one that could be called via via XBMC's built-in commands so that it can be accessed from both python and the HTTP API)

:nod:

[phunkyfish]

Good idea; a "lock_database" and "release_database" mechanism for each database sounds like logical function to have, (one that could be called via via XBMC's built-in commands so that it can be accessed from both python and the HTTP API)

:nod:

In fact without it you may have to reset each XBox back to the start screen (or somewhere) before running a "sync" with other boxes... Not sure about the locking or how it works in XMBC.

[phunkyfish]

Do you think that XBMC would benifit from using ADOdb (or better yet ADOdb Lite)? ???
http://en.wikipedia.org/wiki/ADOdb
http://en.wikipedia.org/wiki/ADOdb_Lite

From what I understand that would not only make it simpler for XBMC to change the database backend if needed (if someone for example like to fork XBMC Linux port to use MySQL instead of SQLite), but it should also more importantly enable better direct access for the Web Interface (HTTP API) and Python to the database?

Yes, generalisation is almost always the way to go. Plus I'd imagine that in XBMC Linux that will be something that will be wanted. From a Java standpoint it is no different as ADOdb is a PHP/Python library. JDBC is Java's equivalent library/interface. The transaction support is a major plus.

A JDBC driver for ADOdb would be nice, but may be pointless, I'll look into it...

[OsirisAD]

I know that this wouldn't be the most graceful solution, but how difficult would it be to make XBMC scrape an XML export from MeD? Couldn't it theoretically scrape the XML for all media items and parse out the appropriate data based on the "Title" field or something similar?

I know this would require an export from MeD and an XBMC resync with each addition, but it seems like it could work... in my head at least.

[RockDawg]

Is this project dead?

[phunkyfish]

Nope!

I've just been very busy!!!

Due to not having enough time to get everything finished off to my liking I'm going to release a beta, it will be very rudimentary and will have almost no UI. It will simply backup the library from one XBox and update it on another. But it will give you something to play with...

I've discarded the idea of using MeD's and will just use the files from XBMC and a Java servlet. So you use XBMC to scrape the IMDB info and then populate it on the other XBox's...

Here's a schedule:

1) Rudimentary App (1-2 weeks, I'm going on Holidays!)
2) Backup and Update App with proper UI (1-2 months)
3) App with Library Viewer (1-2 months+)
4) App with Library Editor (???)

[phunkyfish]

Is the scraping done on XBMC using Python Scripts?

I've recently been using Jython (Python built on Java) quite a bit in work. If the Python script is compatible there's no reason why scraping could not be done via the servlet also. This could give MeD's type functionality to the app???

If it is done in Python is the database functionality done in a separate module? It would be nice if it was as I would only have to rewrite this single module in Jython?

This would of course happen quite a bit into the future but holds a lot of promise...

[Gamester17]

Is the scraping done on XBMC using Python Scripts?No, it is a own module based on Regular expression (Regex) using XML.

I tried to explain the advantages of it here in my suggestion to MeD:
Use XBMC's XML scrapers for HTTP scraping - RFE:
http://sourceforge.net/forum/message.php?msg_id=4553533
Use XBMC's XML scrapers for HTTP scraping - Forum discussion:
http://sourceforge.net/tracker/index.php?func=detail&aid=1807913&group_id=83511&atid=569698

You can download the XML scraper examples here (IMDb.xml is the best one to use as a reference):
http://xbmc.svn.sourceforge.net/viewvc/xbmc/branches/linuxport/XBMC/system/scrapers/video/

More about the scraper function in XBMC can be found in out wiki:
http://xbmc.org/wiki/?title=Scraper
http://xbmc.org/wiki/?title=How_To_Write_Media_Info_Scrapers
http://xbmc.org/wiki/?title=Scrap (scrap.exe to test scapers under Windows for development)
http://xbmc.org/wiki/?title=How_To_use_Scrapers

[phunkyfish]

Cool, OK, I'll port across to Java in the future. Let's see can we get a decent servlet up and running first...

[spiff]

NOOO, i dont want to see my brain child bastardized by java :P (almost j/k ;))

[phunkyfish]

Why not, everything else has at this point??? It's inevitable:laugh:

[spiff]

1) dont take me seriously.

2) i absolutely loath java. it's a horrible language, it's fostering an army of horrible cut'n'paste "coders", its slow bloated and so on. you've heard it all before. but see 1)

[phunkyfish]

You might be surprised to hear I agree with you, at least the army of cut and pasters part anyway (an army of sub standard programmers might be more apt?), the slow, bloated part maybe not.

I think Java has it's place, the standards based specifications and frameworks mean that software is more extendible, scalable and interoperable, that, combined with the platform independence is where it shines.

I was a C++ programmer for many years (and Java before that in my youth) before switching over to Java completely. For web development, data structure and algorithm analysis I think it beats C++. Java's good for some stuff, C++ for others...

To be honest I like them both, but I've made the move to Java and that is probably where I'll stay...

At the risk of this thread becoming a Battle Royale between Java and C++, I'll stop ;)

[spiff]

yeh, sub standard is what i was aiming at. and i also agree, java has its place. just like python has its place. yet, both are horribly misused for just about everything from high performance computing (sigh!) to games etc.and java with its enormous runtime library is obviously a great tool if wielded by someone who knows what he's on about. personally i hate the overly verbose syntax / naming practices. and i will always feel confined when i have to run in a vritual machine, if i want to be nasty to my memory let me damn it. ah well, i'll stop here. my original intent was just to drop a pun so mission accomplished ;)

[aztecnology]

Was just curious if this thread died on the vine, or if the functionality is in a latest release...? I'm setting up my first xbmc/xbox this week and have plans to add at least 1 more soon after...

[Gamester17]

Was just curious if this thread died on the vine, or if the functionality is in a latest release...?It died, no release. Instead see: http://xbmc.org/forum/tags.php?tag=media+manager

[phunkyfish]

Hi guys,

Sorry about there being no release. I got the opportunity to go travelling a year and a half ago and took it. Proud to say I managed to spend only a couple of hours online in all that time.

Switched on my old laptop tonight and in comes this message!

I still have the app I wrote somewhere I'm sure. Has someone come up with a solution in the meantime or might my stuff still be useful?

Now, if I can only remember how to write code ;)

I can barely type, well out of practice...

Let me know,

P.

[Gamester17]

Has someone come up with a solution in the meantime or might my stuff still be useful?No solution yet, but it is still being activly discussed, see:
http://xbmc.org/forum/tags.php?tag=odbc
in particularly in this thread:
http://xbmc.org/forum/showthread.php?t=37449

However this development project might be more interesting for you to join now:
http://xbmc.org/forum/showthread.php?t=50388

You see, while you where away MeD's Movie Manager have become rather out-dated compared to all the other media manager applications that are available for XBMC today, see:
http://xbmc.org/forum/tags.php?tag=media+manager

:;):

[Unbehagen]

Hi,
I don't know if there is an easier alternative to import XBMC's database into MeD's, but I just wrote a small python script to convert the most important data (title, imdb id, file path) to a CSV readable by MeD's.
Here it is:

#!/bin/python
import re,sys
from xml.sax.saxutils import unescape

def getElems(fc,elem):
a=re.findall("<"+elem+">(.+?)</"+elem+">",fc,re.M|re.DOTALL)
return a

def getElem(fc,elem):
a=getElems(fc,elem)
if len(a)>0:
return unescape(a[0], {"&apos;": "'", "&quot;": '"'})
else:
return None

def printHeader():
print '"title","imdb","location"'

def correctPath(path):
if path.find("stack://")>-1:
path=path.replace("stack://","")
path=path.replace(" , ","*")
path=path.replace(",,",",")
return path

if len(sys.argv) !=2 :
sys.stderr.write("usage: "+sys.argv[0]+" path/to/videodb.xml [> output.csv]\r\n")
sys.exit(1)

printHeader()
f=open(sys.argv[1],"r")
fc=f.read()
movies=getElems(fc,"movie")
for movie in movies:
title=getElem(movie,"title").replace("\"","'")
imdb=getElem(movie,"id")[2:]
path=correctPath(getElem(movie,"filenameandpath"))
print '"'+title+'","'+imdb+'","'+path+'"'


Install python, export your database to a single file and start the following from the command line (assuming you are in the directory of videodb.xml and have python in your path):
python xml2csv.py videodb.xml > videodb.csv

In MeD's, select import from csv, select UTF-8 encoding and import your file. Important: Select the option NOT to query IMDB for information. Otherwise it'll search imdb for every movie's title.
In the next screen with the spreadsheet, right-click the columns to assign "IMDB", "Title" and "File Location" to the fields. Import everything.
Select tools->Update IMDB info. Select everything you want to fetch from IMDB (I unchecked title and AKA, because they both(!) overwrite the title I renamed the movie to). It takes a while to re-fetch all information from IMDB, but at least you don't have to re-select every movie in you collection.

Hope this works for you too!