PDA

View Full Version : Two video library suggestions: chapters and multiscraper


pko66
2008-08-20, 13:06
Suggestion 1: It could be nice to have chapters for files that do not have the capability (or to change the chapters on files that does have it, like mkv and such). That chapters info should be incorporated into the database and so could be filled from scrapers or from nfo files.

One possible source of "chapters" is mrskin.com, to have information about female nudity in movies... before you switch on the "pervert alert" alarm, think that the chaptering info could be use to *avoid* the nudity! when the raunchy scene starts, just press "next chapter" and it jumps just after the "thing". Some movies will be really short that way...

Suggestion 2: Multiscrapers

This is important specially for us in the non-english language world, but can also be a nice addition to anyone. For spanish movies ("spanish" meaning here both movies filmed with spanish language and also movies dubbed/subtitled in spanish) there are two scrapers, culturalia and filmaffinity; the catch is that culturalia does not offer thumbnails since the host does not allow it, so if you use it you have no thumbnails; filmaffinity offers the cover art but the quality is much lower than IMDB. Also, usually lists only a few actors, less than in IMDB. And if you use IMDB the movie is usually not correctly identified because of the the translated title, and of course the plot etc is in english.

So, the idea is this: in each directory, some "advanced" button in "set content" should make possible to add additional scrapers and the order to use them. Each scraper could be "additional" (use it to get additional info) or "supplementary" (use it if you have not found the movie in previous scrapers). One possible configuration for movies could be:
1.- Culturalia
2.- Filmaffinity [supplementary]
3.- IMDB [additional]

The scrapper behavior would be:
- Get the info from culturalia. If found, will get the plot in spanish and actors etc info, but no thumbnail
- If you have not found the movie in culturalia, get it from filmaffinity (if found in culturalia, skip this)
- get additional info (actors, thumbnail if not found before...) from imdb

This way people could do partial scrapers; for example, a scraper to get the boxart from movieposters and the chapters from mrskin, and so the order could be:
1.- movieposters
2.- mrskin [additional]
3.- IMDB [additional]

and the behavior would be:
- get info from movieposters (only the thumbnail will be found there)
- get info from mrskin (will get only the chapters)
- get info from IMDB (will get all the info, even the thumbnail if it has not been located in movieposters)

Right now, a scraper must get all the information from a site, but with this, a scraper can be made to get just something (like the thumbnails from movieposters) and be then combined by the user as he/she pleases.

BTW, "local nfo files/local tbn files" could be used as "scrapers" too, to give them the priority desired or to supplement the information found there; right now, if you want to get some extra info from internet for a movie with a nfo file, you must edit manually or delete the nfo file.

The scrapers, when scanning automatically ("scan for new content") will all choose the default movie, so there is the possibility to end with info mixed from different movies from the different scrapers... bad luck!. But since 90% of the time the scrapers get the right movie by default I think that is not a big problem. To correct that, when getting info in manual mode, for each scraper it will show all the possible movies found just like they do now, but in addition to the "manual" button a "skip" one would appear and, when pressed, it will skip the current scraper and proceed to the next it the chain. For the thumbnail selection, all the thumbnails found in all the scrapers should be shown to the user to select.

Also, another GUI change should be added to the "movie info" window; now you have the "refresh button" that simply re-scrapes the movie with the defaults in the directory. Perhaps with a new dialog appearing when selecting "refresh" or perhaps with a new "advanced refresh" button, the user could opt to:
- re-scrape the movie with the default options from the directory (current behavior)
- re-scrape the movie from scratch (deleting previous info) with the scraper of choice
- re-scrape the movie to get new information in addition to the one already present
The local nfo/tbn files would be just one scraper more to choose from.

One final consideration; when mixing info from different sources, each field should have its proper treatment (although perhaps that treatment could be changed in advancedsettings.xml):
- most (plot etc) should be skipped if the info already exists
- some (thumbnail) should be added to give the option to choose
- some (actors) should be added, but avoiding duplication of entries that already exist

spiff
2008-08-20, 13:14
too long a post ;)

i have been thinking about chaining scrapers, but for now you can pick which part you want by editing. you can chain to any website you like

pko66
2008-08-20, 15:23
too long a post ;)

Yes, I thought of that... AFTER writing it! :laugh: sorry!

i have been thinking about chaining scrapers, but for now you can pick which part you want by editing. you can chain to any website you like

Yes, but I think that 1) you have to modify the code, and if you need data from more than one site, you are given the option to choose a specific movie just once, aren't you? 2) when you "refresh" the data, all the old fields get erased, right? so the scraper must read *all* the fields from the net...

Good to know it has been at least considered :grin: I think that it has a great potential

What about my other suggestion, incorporating chapter info in to the videodb to later use it in playback? even if it can be only read from nfo files can be useful, tools like the XBMC Media Companion by billyad2000 could be used to create the appropriate XML tags!

Wow! again writing a too long message... :oo:

spiff
2008-08-20, 16:32
sure, after atlantis it's certainly something that can be considered

Gamester17
2008-08-20, 16:52
Suggestion 1: It could be nice to have chapters for files that do not have the capabilityXBMC already support SceneMarkers and EDL (Edit Decision List),see:
http://xbmc.org/wiki/?title=EDL_%28commercial_skipping%29_and_SceneMark er_support

or to change the chapters on files that does have it, like mkv and such.That you will have to continue to do on your computer, XBMC does not touch 'tags' in any files. Either that or use the above mentioned SceneMarker files.

chapters info should be incorporated into the database and so could be filled from scrapers or from nfo files.I do not see why chapter information need to be imported into the database/library, XBMC only ever use them when playing the file in which case it reads the SceneMarker files and chapter tags in the container anyway. So chapters can be read on-the-fly whether it be from a SceneMarker file or from containers like MKV and DVD-Video which have integrated support for chapters. Again see: http://xbmc.org/wiki/?title=EDL_%28commercial_skipping%29_and_SceneMark er_support

One possible source of "chapters" is mrskin.com, to have information about female nudity in movies.You could write a python script that downloads those and convert them to a SceneMarker format which XBMC supports, (similar to the XBMCLyrics (http://xbmc.org/forum/showthread.php?t=10187) or OpenSubtitles (http://xbmc.org/forum/showthread.php?t=31499)scripts).


This is important specially for us in the non-english language world, but can also be a nice addition to anyone. For spanish movies ("spanish" meaning here both movies filmed with spanish language and also movies dubbed/subtitled in spanish) there are two scrapers, culturalia and filmaffinity; the catch is that culturalia does not offer thumbnails since the host does not allow it, so if you use it you have no thumbnails; filmaffinity offers the cover art but the quality is much lower than IMDB.Like spiff said, you can already today chain lookups internally inside a scraper, the IMDb scraper can for example also download even higher quality posters/thumbnails from impawards.com and movieposter.com (as an optional setting the user can activate when setting content for a source), you can read more about that here => http://xbmc.org/forum/showthread.php?t=28321 (or even better, look inside the imdb.xml scraper file).

pko66
2008-08-21, 11:42
XBMC already support SceneMarkers and EDL (Edit Decision List),see:
http://xbmc.org/wiki/?title=EDL_%28commercial_skipping%29_and_SceneMark er_support

Wow! Gamester17, your knowledge of XBMC is simply astounding!

And also, XBMC amazes me once more with its capabilities, never heard of that EDL files...

Will investigate the matter, perhaps I could do some kind of tool to create the EDL files. Cannot be a script, I think, since it would not work over smb:// shares (that are almost all of my video sources). The reason to incorporate the chapter information to the videdb was precisely that... there is no directory structure "a la thumbails" to create thos EDL files, isn't it?

That you will have to continue to do on your computer, XBMC does not touch 'tags' in any files. Either that or use the above mentioned SceneMarker files.

Of course, the idea was not changing the source files!!! it was to create somehow the chapter information in a way that would have higher priority to XBMC

I do not see why chapter information need to be imported into the database/library, XBMC only ever use them when playing the file in which case it reads the SceneMarker files and chapter tags in the container anyway.

The problem is that a script cannot write to most sources outside the local hard disks, and there is also the problem with read-only sources like DVD-R; there the only solution is videodb or an alternate directory like thumbnails or subtitles. But for my purposes, a tool that creates the EDL files under windows or linux (like XBMC Media Companion creates NFOs and TBNs) will do.

Like spiff said, you can already today chain lookups internally inside a scraper, the IMDb scraper can for example also download even higher quality posters/thumbnails from impawards.com and movieposter.com

I plan to study both python and scraper creation (thinking of creating a documentary scraper), and this can be a good starting point, the problem I see is that (AFAIK) a scraper can ask for the correct movie just once; that is no problem when you can use the IMDB ID like I think is the case with impawards or movieposter, but culturalia or filmaffinity (where the spanish language plots are obtained) have different structure; user must confirm at least twice (more if the movie is not in the database) and right now you cannot do that with a scraper, do you?

Thank you very much for your enlightenment, Gamester17, maybe you should be called "the XBMC guy" or "the XBMC guru" just like Forrest J. Ackerman gets sometimes credited as "the SF guy" :-D.

KidKiwi
2008-08-22, 03:11
I'd like to see where this leads. I made a post about a similiar (Maybe the same) thing here http://xbmc.org/forum/showthread.php?t=35991.

I realised later that a file - like the "Chapter.txt" file a lot of programs spit out - would probably be the way to go. I'm often wrong about these sorts of things, so I always follow feedback/discussions as best I can.

kraqh3d
2008-08-23, 07:38
the support is already in xbmc. really, any number of "chapter" layout files could be supported, provided details exist on how to parse them. the issue with chapters.txt is that its too generic of a filename. many users have lots of videos in a single folder. in this case, which video would that correspond to?

quebert
2008-09-30, 02:49
I'm an XBMC n00b, so far it hasn't really been too hard too figure things out, but I too am interested in chapters. I read up on EDL & Scenemarkers and neither seem to be what I'm looking for exactly. I would like chapters that can be used in my video menu. For example I rip my UFC DVD's, I can either rip fight by fight and you have to select which one to watch and it can't be seamlessly played from the start to the end, or I could rip it into 1 big AVI and it has to be watched from the start to the end. With Scenemarkers I could put previous/next points in the video, but not have them be able to show up on the menu right? As it stands I have about 20 of my UFC DVD's ripped to 1 large AVI and would LOOOOVE to have each PPV broken up with an entry in my video library menu. Like each PPV was a season of a show, and each fight was an episode in that season.

Does this make any sense? I think it's close to what the OP is asking for or at least related :)

KidKiwi
2008-09-30, 06:03
If they were ripped as single episodes you could string them together with a playlist.

quebert
2008-09-30, 07:42
KidKiwi it would, but that would require a deal of work too. Wouldn't you have to basically create a playlist for each entry? You could duplicate them and edit them I suppose but it still seems like extra work. And even if it's more simple than I'm figuring it to be, I have 20 DVD's ripped into 20 single AVI's, I *could* edit them and split them apart - now that would take some time :D

pko66
2008-09-30, 12:25
Does this make any sense? I think it's close to what the OP is asking for or at least related :)

Well it *should* work right now, although I tried and failed... it was just a quick test. Supposedly, you just create a text file with the same name of the video file and extension .edl (just like subtitle files). The format of the EDL file is simple, just see this entry (http://xbmc.org/wiki/?title=EDL_(commercial_skipping)_and_SceneMarker_s upport) in the wiki manual. You must also (in video settings) activate the option for EDL and modify keymap.xml to assign a key for the jumps.

If you do that, just create a EDL file; let's suppose that your UFC is named "UFC_001.avi" and that that contains three fights that start at 45 seconds, 321 seconds and 723 seconds from the beginning. Then you just create a plain text file named "UFC_001.edl" in the same directory of the video file with this content:

1 45 2
2 321 2
3 723 2

if you assigned the skipplus key to NextScene in keymap.xml, you could just press it to jump to the next fight. I tried it and didn't work, but was just a quick test (for lack of time) and probably I made some mistake. I haven't tried again (I simply forgot about it!), please tell here your results if you try.

quebert
2008-09-30, 19:56
hummm pko66 I couldn't get it to work either, even if I could wouldn't it still only show the one AVI in my library? Your work around would be great as an alternative and at least allow skipping, but I think it would still not allow for individual entries in my library. I will mess around with what you suggested and see what I can come up with, thanks :)

pko66
2008-09-30, 20:36
hummm pko66 I couldn't get it to work either,

So maybe this functionality is broken... should be "trak"ed then.

even if I could wouldn't it still only show the one AVI in my library? Your work around would be great as an alternative and at least allow skipping, but I think it would still not allow for individual entries in my library. I will mess around with what you suggested and see what I can come up with, thanks :)

No, of course... if I'm not mistaken, you need a file (or more) for each entry, for example double episodes of tvshows get just one entry in the videodb.

KidKiwi
2008-10-01, 00:34
All this sounds like a lot of work! My original idea was that you could do this from the GUI - cycle through the video and select a "skip" point (like using the bookmark area). I'd like to see this use both DVD chapters and these added "skip" points, so if a DVD didn't have enough chapters, then you could add a few more skip points via the GUI and they work concurrently.

If this is not working for you, you can always open the DVD rip in Vobblanker (free) and split the cell (Using the "Add program and Chapter" check boxes) then export. This will add new chapters to your video. You can even backup the IFO/BUP files to drop over the originals if things get lost. This way you don't have to do all the work again. Ultimately, I think having a media system setup the way we all individually like it is a lot of work :D