View Full Version : Make library mode work without scrapers - import all files into the database
I would like to suggest a new option so that library mode would work without using scraper/nfo files. I believe that both scrapers and .nfo-files should be optional.
If no scraper is selected, and no .nfo file is available. Then a very basic library would be created using the name of the folder (or the name of the file, based on that setting).
I download new movies all the time. I delete movies all the time. Most of my movies are non american (and cannot be found by any of the scrapers). In my situation it is impossible to use the library mode. I just dont have the time to manually create .nfo files for all my movies.
I'm sure that there are a lot of people having the same problem as I do.
Feedback?
I think this would be a very valid option for something like home movies (which you could use for movies if you want to).
It should be relatively easy to implement (all the code is already in place for Music Videos).
It's just a matter of convincing a dev it's something he really needs.
If you don't use Music Videos in your library you could play with that (NFO scraper) and see how it goes.
Maby the system doesn't need to be altered at all. Maby the only thing that is needed is a new scraper that "scrapes" info from folders/file names. It could be called "Folder scraper" or "Filename scraper". Or "Default scraper" that looks at the setting "use foldernames when scraping".
I might be wrong here since I'm not a developer, and don't know anything about the inner workings of scrapers.
what's the point of a library without any info? we already have that - it's called file view
The point is that
1) lots of skins (aeon, mediastream) doesn't fully support file view. The new XBMC remote for iPhone doesn't support file view. Using library seems mandatory.
2) there should be a easy way to get started with the library without using scrapers and .nfo files. In my case scrapers doesn't find anything and .nfo takes to much time to created.
With a "default scraper" that use filename/folders I can start out easy - and choose to put more info in some of the .nfo files if I want to. Or I can choose to use a more advanced scraper for some of my movies.
Using a advanced scraper for all of my movies is not a working solution for me. It will only give me invalid data. I want to use a filename/folder scraper first, and then move on to more advanced scrapers/nfo-files later if I feel like it.
Think of this feature as a way for Library newbies to start using the library.
Sorry for tripple posting. But I just wanted to say that I'm not the only one with this problem. This Nfo generator was created for beginners that want to start using the library http://xbmc.org/forum/showthread.php?t=36167
However. My suggested solution seems more elegant than using a platform dependant software to generate these nfo files.
nothing is elegant about voiding the entire point of a thing. your solution is no solution at all.
Sorry to hear that. What do you believe is the best way for xbmc beginners to start using the library if they are in the same situation like me (1000s of foreign movies)?
where are those movies from? reason i ask if so we can make a scraper to cover them if a suitable backend is available.
Gamester17
2008-09-16, 13:33
@everyone, please read through this other thread which discusses something very similar:
http://xbmc.org/forum/showthread.php?t=14186
Also checkout this other related idea: http://xbmc.org/forum/showthread.php?t=33710
I believe that MeediOS (http://www.meedios.com) does something like this(?), ...I have not used MeediOS (http://www.meedios.com) myself but I understand that they do not even have a 'file-mode', instead they scan in all file-names into the database and just display the file-name in library-mode if no meta data could be found online by their scrapers, (MeediOS (http://www.meedios.com) call their 'scrapers' for "importers" by the way, but their purpose is the same as the scrapers in XBMC).
Some are english, some are swedish, but most are very short videos created by myself and have private content (family etc.).
The short videos don't have a genre/actors etc. They only have a name.
Some of the swedish movies can be found in imdb, but not all (a new scraper might solve that)
But the main problem for me is that I want to list all movies (including my home videos) in library view so that I can view them in Aeon/MediaStream/XBMC Remote for iPhone.
It seems to me that creating a new scraper is the best solution sometimes, but not always.
If you want everyone to start using the library, then you'll have to make the library really easy to use. A default scraper using filenames/folders is a good start in that direction. It will always work. Using the IMDB or another scraper doesn't always work - and that complicates things. Average joe just want things to work ASAP - without issues.
I have not used MeediOS (http://www.meedios.com) myself but I understand that they do not even have a 'file-mode', instead they scan in all file-names into the database and just display the file-name in library-mode if no meta data could be found online by their scrapers
That seems logic (even if it might upset some users). Having both file mode and library mode is only confusing. At least for beginners.
see, what you actually want is a file node in library. not to scan all vids into the db
see, what you actually want is a file node in library. not to scan all vids into the db
Incorrect.
If I want to I should be able to add more info to my home videos (like a description or a picture). So I need them all to be in the db. File view is not sufficient.
Gamester17
2008-09-16, 14:04
see, what you actually want is a file node in library. not to scan all vids into the dbWell, not exactly, if all files where scanned into the database first on file/folder name alone, (before going online to look for meta data for it), then the user could still use the library search function and live search to find files. Without a database that stores all the file/folder names searching is not exactly usable if no metadata for the item was found online.
I touched this discussion with the MeediOS developers when I suggested that they implement XBMC's scraper parser into MeediOS so that we could share that code, and there I learned that MeediOS process flow is slightly different than XBMC.
http://www.meedios.com/forum/viewtopic.php?t=2238
Currently the process flow is like this in XBMC:
File/Folder-name => Regular Expression Importer Filter => Media Meta Data from XBMC Scrapers => New database entry
While with using MeediOS concept the process flow in XBMC would be a little more like this:
File/Folder-name => Regular Expression Importer Filter => New database entry => Extra Media Meta Data from XBMC Scrapers
I non XBMC developers do not know; what the "Regular Expression Importer Filter" does is try to cleanup the name before preforming the lookup, like remove the file-extension, replace dots with spaces, assume that four digests in a parentheses is the year it was produced, remove some known 'scene ripper' tags such as "xvid", "x264", "dvdrip", "dvdscr" and such.
Again, please read thoroughly through this related discussion if you are truly interested in what was said previously about this idea as it really depends on how easy we can first make it for new users to add sources to XBMC: http://xbmc.org/forum/showthread.php?t=14186
If implemented then this should IMHO be the future default way to browse view mode in XBMC :nod:
...sure we would still keep file-mode for those who insists on using but it would no longer be the default.
While with using MeediOS concept the process flow in XBMC would be a little more like this:
File/Folder-name => Regular Expression Importer Filter => New database entry => Extra Media Meta Data from XBMC Scrapers
Does your suggestion imply that XBMC scrapers are mandatory. I feel that XBMC scraping should be optional. I don't want to scrape all my movies since I already know that nothing good will come from it.
May I also suggest that you add the name to the .nfo-file, so that it will be easy to add new information manually (with XBMC Media Companion/notepad/...). AFAIK there is no software available that modifys the movie/tv-series information in the XBMC database directly.
Gamester17
2008-09-16, 15:50
While with using MeediOS concept the process flow in XBMC would be a little more like this:
File/Folder-name => Regular Expression Importer Filter => New database entry => Extra Media Meta Data from XBMC Scrapers Does your suggestion imply that XBMC scrapers are mandatory.Not exactly, for it to be totally automatic with no user intervention and as user-friendly as possible for the majority of the XBMC userbase; yes all files would be scraped, but not like you seem to think.
File/Folder-name => Regular Expression Importer Filter => New database entry => Extra Media Meta Data from XBMC Scrapers
How this concept would work in practice is that first all files are scanned and each file generate an new entry item in the SQL databese, (and it should be noted that all that will be done in the background so that the user can still use XBMC while all the scanning is happening), this entry item would always contain the location of the file as well as the full file name (which is part of the location of course), it would in this scenario also contain the file-extension (so that it can be filtered/sorted out separately by the end-user if they which) and a name generated by the Regular Expression Importer Filter parser, and all this is done before XBMC goes online with the scrapers to look for additional meta data for this item.
For example:
The user set a source as movies and choose whichever movie scraper for it (as they still need to pick one, in this example we say IMDb), then XBMC starts scanning all files in that source into the database under the category "Movies".
The scanner finds a file named "Pulp.Fiction.(1994).XviD.DVDrip.avi" in that source, it enters the location path including the full file-name into database but it also runs the Regular Expression Importer Filter which cleans up the name it enters for than database entry as only "Pulp Fiction", this because the Regular Expression Importer Filter would parse the name to clean it up, entering the year for that item into the database (as it assume that four digests in a parentheses is the year it was produced), also replacing the dots with spaces and remove the known 'scene ripper' tags which in this example is "xvid" and "dvdrip", also the extension is stored so that the user can still sort by extension. Now if the end-user had an XML formatted NFO for that file then all the meta data from the XML formatted NFO would also have been entered into the database with the Regular Expression Importer Filter parser.
So far so good, our file named "Pulp.Fiction.(1994).XviD.DVDrip.avi" have been entered into the database, (and all that before going online yet), so if the user did not have the internet they would now still be able to search and browse it in the library more.
Now comes the next step which is done in a separate process thread; again, all done in the background without the user really noticing it as they will still be able to use XBMC as normal when all this is going on, being the scene so to speak; XBMC automatically beings to take those parsed names from the database that was generated with the help of the Regular Expression Importer Filter and send them to the IMDb movie scraper and it goes online with the name, and year if available, (note that XBMC now take those names and years from the SQL database entry, not from the file-name), the scraper tries to find a match online, if it does not find a match online then nothing more happens and you are still left with the name generated by Regular Expression Importer Filter to use in the library view mode, but if the scraper does find a match online which is would in our example with "Pulp Fiction" then it will append the meta data it collects to the database entry for that item, and it will also overwrite the initial name generated by the Regular Expression Importer Filter, (that is unless that name was picked up from an XML formatted NFO file, in which case it would use that name instead).
May I also suggest that you add the name to the .nfo-file, so that it will be easy to add new information manually (with XBMC Media Companion/notepad/...)The name and location of the .NFO files does not need to be in the database as the location and full name of the file is always also already in the database, and the NFO file always have the same name as the file-name but with a .NFO as the extension instead. If you make any changes inside the NFO file then XBMC will pick that up on the scan refresh, ...as far as I know XBMC does this already.
Thank you for your explanation. Very interesting! It seems as if this is a small modification to how XBMC works today!
if it does not find a match online then nothing more happens and you are still left with the name generated by Regular Expression Importer Filter to use in the library view mode
What if there is a match, but the match is not the correct movie. Is there a way to remove or edit the incorrect data?
And what happens if there are several matches?
The name and location of the .NFO files does not need to be in the database as the location and full name of the file is always also already in the database, and the NFO file always have the same name as the file-name but with a .NFO as the extension instead. If you make any changes inside the NFO file then XBMC will pick that up on the scan refresh, ...as far as I know XBMC does this already.
I was referring to the name of the movie. Not the name of the .NFO file. What I mean is that it should be easy to edit the information about the movie, and that the .NFO should contain the information (name of the movie) returned by the Regular Expression Importer Filter.
Gamester17
2008-09-16, 16:27
Note that this is all a concept, not something that are in the works or even planned yet. We are still just brainstorming openly here, no one has gotten any closer to the dawning table.
if it does not find a match online then nothing more happens and you are still left with the name generated by Regular Expression Importer Filter to use in the library view modeWhat if there is a match, but the match is not the correct movie. Is there a way to remove or edit the incorrect data?Yes functions would have to be implemented so that you could choose only to use the Regular Expression Importer Filter generated name, and to edit that name directly from the XBMC GUI. Wrong matches is always going to happen and is unavoidable to get rid of completely. What will make it easier is to have more genres/subgenres in scrapers, video database, and library filters, some of which (like for example "Home Movies") will not be connected to a scraper (and only use NFO files if available).
http://xbmc.org/forum/showthread.php?t=33710
what happens if there are several matches?Same as today, (when you do a scan today on IMDB movie names get multiple hits), when scanning an entire directory XBMC will pick the top hit, it would be annoying if XBMC stopped to ask on every multiple hit. Again, mismatches will happen, the user should be able to go in after and current them, manually or by doing a new individual online lookup when you get the option to pick the corrent one on multiple hits just like today.
The name and location of the .NFO files does not need to be in the database as the location and full name of the file is always also already in the database, and the NFO file always have the same name as the file-name but with a .NFO as the extension instead. If you make any changes inside the NFO file then XBMC will pick that up on the scan refresh, ...as far as I know XBMC does this already.I was referring to the name of the movie. Not the name of the .NFO file. What I mean is that it should be easy to edit the information about the movie, and that the .NFO should contain the information (name of the movie) returned by the Regular Expression Importer Filter.[/QUOTE]That is already how it works today in XBMC so that would not change, any strings that exist in a XML formatted NFO is prioritized over any meta data collected online with a scraper for that specific string, see: http://xbmc.org/wiki/?title=NFO
Maby the system doesn't need to be altered at all. Maby the only thing that is needed is a new scraper that "scrapes" info from folders/file names. It could be called "Folder scraper" or "Filename scraper". Or "Default scraper" that looks at the setting "use foldernames when scraping".
I might be wrong here since I'm not a developer, and don't know anything about the inner workings of scrapers.Did you read my post?
There is something very close to such a thing, it's called Music video NFO scraper.
If a file does not have an associated NFO file it will use the filename. (xxx - yyy may be required).
If you don't have a problem with them being called Music Videos, this could be exactly what you are looking for?
the way to proceed to add this EVIL (imo) "functionality" would indeed be to extend the filename scraping to be supported for movie content
I for one would love to have a "home movies" content type with a filename/nfo based scraper.
I had just decided on waiting for post 8.04 to start badgering the devs ;)
TeknoJnky
2008-09-23, 01:21
I really like Gamester17's idea (http://xbmc.org/forum/showpost.php?p=220703&postcount=18) and hope it is something that would get implemented in the future.
If it would be easier/quicker to implement, allowing the 'music video' scraper to be utilzed as a fall back for when IMDB or other selected scraper fails, might be a usable alternative.
the way to proceed to add this EVIL (imo) "functionality" would indeed be to extend the filename scraping to be supported for movie content
I am not sure why you think this is evil. I think this would be a good step into getting people to who use file mode, into trying/using library mode... especially in content situations as likened by markiz.
rwparris2
2008-09-23, 19:28
I tried searching but is there a request in trac for this yet?
If not I'll make one
why i think it's evil? because it isnt a solution at all - it is just cluttering your library.
the solution is to make the library SUPPORT the content type you want in there.
TeknoJnky
2008-09-23, 19:55
why i think it's evil? because it isnt a solution at all - it is just cluttering your library.
the solution is to make the library SUPPORT the content type you want in there.
Yes, but the problem appears to be, there can never be any (external) scrapers for custom/home/private content... yet this same content, the user may desire to have within the same library context as other publicly accessible media. IE home movies mixed with hollywood movies.
Besides, clutter is in the eye of the user. While some may consider home movies (or other content with no real metadata) mixed in with regular movies to be clutter, others may not.
sure. you asked me for my view. i gave you that.
rwparris2
2008-09-23, 20:19
Besides, clutter is in the eye of the user. While some may consider home movies (or other content with no real metadata) mixed in with regular movies to be clutter, others may not.
A better solution IMO is custom categories. Having Home Movies mixed with 'Real' Movies yet having TV Shows & Music Videos separated is inconsistent, even if you don't consider it clutter.
I created a ticket asking for custom categories here: http://xbmc.org/trac/ticket/4969
TeknoJnky
2008-09-23, 22:48
A better solution IMO is custom categories. Having Home Movies mixed with 'Real' Movies yet having TV Shows & Music Videos separated is inconsistent, even if you don't consider it clutter.
I created a ticket asking for custom categories here: http://xbmc.org/trac/ticket/4969
Good idea, for my own use, I would rather have all videos in one node and use filters when I want to show only 1 category.
Further, it would be sweet you could color code the categories so that they appeared differently while in an 'all videos' view.
rwparris2
2008-09-24, 00:54
Good idea, for my own use, I would rather have all videos in one node and use filters when I want to show only 1 category.
That could be done with my proposed feature, however at this point I have to wonder why not just use file mode?
Gamester17
2008-09-24, 02:33
A better solution IMO is custom categories. Having Home Movies mixed with 'Real' Movies yet having TV Shows & Music Videos separated is inconsistent, even if you don't consider it clutter.
I created a ticket asking for custom categories here: http://xbmc.org/trac/ticket/4969http://xbmc.org/forum/showthread.php?t=33710
:;):
rwparris2
2008-09-24, 06:47
http://xbmc.org/forum/showthread.php?t=33710
:;):
I forgot about that! Should have done a search... anyways, thanks for adding a link to the trac ticket for me.