Get multiple entries in library for stacked files after update (imdb-scraper)

[ForgotMyUsername]

Hiya....

I updated to the latest SVN today and at first everything seemed to work fine. I then updated my videolibrary to scan for newly added content. It also worked fine with the exception that it now scans all stacked files as new content, so I have three entries in the database for each stacked movie. One for the stacked name, one for the first CD and one for the second. My first suspicion was that recursive scanning could be on, but it isn't. As I backed up my library I put it back on the box and tried to set up the imdb-scraper again, but with no effect. It keeps scanning stacked content. Any ideas?

Here's the log:

http://pastebin.com/m24c26ba7

ForgotMyUsername

[jmarshall]

The way stacked folders are stored has changed, this is why you are seeing dupes.

You need to remove the old entries (rescanning all if it's not too much trouble).

[SandmanCL]

Is this related to what I'm seeing as well ?
My video library, which I copy from version to version, no looks apparently fine in PMIII's Media Info view. Plot, movie rating, etc. all show up. But when I hit 'Info' on a movie, it looks for it @ imdb again.
When it's done looking, I'm left with two copies. Both play the same files.

If I need to rescan the library again I think it's time to suck it up and create .tbn files for all movies on my NAS :grin: I'm getting tired of manually fixing sucky posters each time.

[ForgotMyUsername]

Tried to delete all multiple entries and did a rescan. Thereby I discovered that the problem is even worse than I originally thought. The stacked files will be added to the database every time I run "Update Database". So now that I ran "Udate database" three times I am left with four entries for each stacked movie in the database. Library works fine for all non-stacked items. Weird...
When I update, the information text on the screen indicates that the scrape is looking up the second CD every time I do a scan. I have no idea what is going wrong here.

[jmarshall]

Please produce a full debug log of this process.

[ForgotMyUsername]

I don't really know what you want me to log, but here's what I did (hope it might help).... I tried to track down the problem by removing all stacked files from my share except for one. This might make things easier.

1.) Tried to delete all database entries for following movie. Filenames:

Der Herr der Ringe - Die Gefährten (2001) (SEE) 219' [AC3-D-E] -CD1.avi
Der Herr der Ringe - Die Gefährten (2001) (SEE) 219' [AC3-D-E] -CD2.avi

also deleted:

Der Herr der Ringe - Die Gefärten (2001) 172' [AC3-D].avi

Unfortunately only two entries were removed, the third one could not be deleted.

Here's the log:

http://pastebin.com/m3b2ab780


2.) Rebooted, updated database, expected to have two entries, but had three entries again.

Here's the log:

http://pastebin.com/m613116ce

Hope there will be a solution, in the meantime I'm joining my stacked files with Virtualdub... seems to be the only thing that really works....

ForgotMyUsername

[jmarshall]

The stack one you couldn't delete could possibly be due to the old way the stack scheme was stored. I suggest you rename your videodb and rescan.

If you think there's a bug regardless, then please simplify down to the smallest possible dataset (eg just 3 files - 2 of them stacked in the database) and produce a log from that, as well as upload the db file.

Cheers,
Jonathan

[ForgotMyUsername]

I'm afraid I don't really get the point. If I rename my videodb and rescan, I could as well delete it and lose 1200 watched-tags. I don't really feel like setting them manually (besides, I don't really remember which movies I watched, that's what I want the database for).

Is there any way to delete the old entry in the database via SQLite Analyzer or anything the like? I'm ready to try anything, clean rescanning is the only thing that is no possible option.

Concerning the minimal setup, I'm sure there is a bug, but I'm also sure this will not occur in a clear setup with a clean database so it would make no sense wasting time on this, because the bug occurs in the circumstances documented in the logs above.

Perhaps you could briefly explain the changes that have been made to stacked files, this could also help to track down the problem. I've searched the forums and the changelog since your last post but couldn't find anything.

ForgotMyUsername

[jmarshall]

There is not a bug if we changed the videodb layout and it works fine with the new layout.

What you are seeing is simply that it doesn't work due to the dead items in the previous layout. We do not guarantee trouble-free updates (though we try and keep disruption to a minimum, sometimes it is unavoidable).

There is very likely a way to remove the items correctly using sqlitespy or whatever.

I suggest you upload your videodb and allow us to take a look in it. You can ofcourse vacuum out any passwords etc. (in path table) via sqlitespy or similar.

Cheers,
Jonathan

[ForgotMyUsername]

Thanks for your reply. Of course I'm aware that there is no guarantee on updates and I really appreciate that recently everything possible has been done to make rescanning unnecessary. But sometimes I still feel like people don't really know that the database is sacrosanct for every user who tries to use xbmc for what it was originally designed for.

Enough philosophy for today, let's try to find the problem. I've xx'ed my passwords and PM'ed you a link to my database. Hope it will help.

ForgotMyUsername.

[jmarshall]

I ran the following queries on your db:

First to find files in the db from the old naming scheme:


select idfile from files on files.idfile=movie.idfile join path on files.idpath=path.idpath where strPath like "stack://%"


To check movies with this scheme:


select * from movie where idfile in (select idfile from files on files.idfile=movie.idfile join path on files.idpath=path.idpath where strPath like "stack://%")


To clean it up, I performed the following:


delete from movie where idfile in (select idfile from files on files.idfile=movie.idfile join path on files.idpath=path.idpath where strPath like "stack://%")
delete from files where idfile in (select idfile from files on files.idfile=movie.idfile join path on files.idpath=path.idpath where strPath like "stack://%")
delete from path where strpath like "stack://%"


Now, there still seems to be dupes for LOTR, so what I suggest you do (after doing the above!) is:


delete from files where idfile in (select idfile from movie where c00 like 'The lord of the rings%')
delete from movie where c00 like 'The lord of the rings%'


Then load up XBMC and do a forced rescan of the DEF folder. That should take care of things.

Cheers,
Jonathan

[SandmanCL]

[quote] [...] That should take care of things.[/quote.

This would make for a fun quote in a thread illustrating how simple it is to work with xbmc :rofl:

(I am joking -- I do fully understand that in order to improve a product one can't always be backwards compatible)

[ForgotMyUsername]

Time for an update...

Tried to run the database operations, but I had an SQL error (-1) with this:

delete from file where idfile in (select idfile from files on files.idfile=movie.idfile join path on files.idpath=path.idpath where strPath like "stack://%")

Put the database on the box. The stacked files were gone but so were all my covers. Then rescanned and it started adding _all_ movies again. Aborted after 2-3 hours because this made no sense.

After thinking about what you explained, I found a solution, however. So I'd just like to share this here in case anybody else may encounter the problem:

HowTo: Remove old stacked items from the database

1.) Downgrade to a version which uses the old stacking scheme. (I used XBMC-SVN_2008-01-06_rev11247-T3CH).
2.) Move all stacked content off your shares (to any path that will not be scanned).
3.) Use Settings/Video/Clean database so that the stacked content gets removed from the db.
4.) Upgrade to lastest SVN and set up scrapers (in my case, scraper-settings for multipaths were lost after the upgrade).
5.) Reboot.
6.) Move your stacked content into shares again.
7.) Do a rescan on all shares.
8.) Your stacked content should now be in the database again and no duplicates will be created when doing scans in the future.

ForgotMyUsername

[SandmanCL]

Thanks, this approach will save me a lot of time. Sometimes the simplest solutions are the ones hard to think of :)

[SandmanCL]

... but actually, why go through the steps of downgrading first ?

This should suffice:

1) move stacked files away from your share
2) clean library
3) move stacked files back to movie share
4) update library

[jmarshall]

It should have been reasonably clear that it was a typo - everywhere else it uses "files" as the table name, and sqlitespy would have made it clear to you. I've fixed the original post so that it hopefully helps others.

It also would not have affected thumbs at all (no reference to thumbs in db), and also would not have affected what was in the db and what wasn't (path hashes). Hence, your paths must have altered in some way - most probably because you used the version of your db that you'd changed the passwords in...

A little thought and you would have been all good, but never mind - you seem to have come up with a solution on your own which is great. Hopefully your experience will help others.

Cheers,
Jonathan

[ForgotMyUsername]

@ SandmanCL: I doubt that this approach will work, because I was not able to remove the stacked items from the database using the context menu. That's why I think a downgrade is necessary as it correctly removes the stacked files from the db. Perhaps you might keep us updated if the easier approach works as well.

@ jmarshall: I also suspected that this might be a typo and tried "files" as well. Of course this did not produce the sql error but still had the effect described above (which only occurred in listview btw).
You might be right in that I could have used the wrong db, but I really can't tell. Anyhow, it's highly unlikely that my server would have allowed the scraper to scan movies 2-3 hours with those xx'ed passwords, usually it should not grant access without correct passwords :shocked:. Unfortunately I can't reproduce it anymore as I deleted both databases a few hours ago. However, I have still have the oldest backup in this series, so I would be able to try it on that one. But I think as long as there aren't more people with the same problem, it's not really worth the time. Let's wait and see whether this thread hopefully dies or if there's a real need for it.

ForgotMyUsername

[jmarshall]

The db would have had the wrong pass's in the path table, but your sources would have had the correct path. As this correct path was no longer in the database, it proceded to go and rescan all these "new" paths in.

Glad you got it working though :)

Cheers,
Jonathan

[SandmanCL]

@ForgotMyUsername: The approach I suggested worked just fine. Upating the library using the context menu will only look for new files, but when you clean the library it will remove missing items.

[spiff]

lies. update library cleans as well

[ForgotMyUsername]

Just to put things right, the thing that did not work was removing files from the database using the context menu on database entries with "remove item/entry" (or whatever it might be called in English, I use a localized version). That's why I did the downgrade because the older version is able to remove the items from the database which the newer version obviously is not able to do (for left-over stacked items from older versions only of course).

As there is no edit-option in the forum, could someone please rename the topic to "SOLVED: Get multiple...."???