advancedsettings.xml cleanonupdate issue

[tret]

Hi guys,

First of all I am running:

Ubuntu Hardy 8.0.4.1
XBMC Atlantis Final

I use sabnzbd+ for downloading media and I wrote a script to run once the download is complete. The script moves the downloaded movie/tv show to my xbmc library location and sends a video library update command to the xbmc web server via this command:


curl --get "http://192.168.1.6:8080/xbmcCmds/xbmcHttp?command=ExecBuiltIn&parameter=XBMC.updatelibrary(video)"


This works great and the video library is updated automatically every time a new movie/tv show is placed in the xbmc library location by my script.

I have this option set in advancedsettings.xml so that any movies/tv shows that I have recently deleted will be removed from the video library automatically whenever my script above updates the video library.


<cleanonupdate>true</cleanonupdate>



unfortunately this clean on update isn't happening. The library update takes place as usual but the library still has entries for videos that no longer exist. If I go to video library settings within XBMC and manually select clean library the entries for the videos that no longer exist are removed successfully.

Does anyone have any idea why the <cleanonupdate> setting isn't taking place? Does anyone else do something similar and have this working properly?

Thanks,
Rob

[spiff]

forgotten the

<advancedsettings> tag?

[tret]

forgotten the

<advancedsettings> tag?

It wasn't this but there was a typo elsewhere in the file. I forgot to close the video section, had <video> but left out the </video> to close it. So the entire file was being ignored, fixed the error and all is well!

[kricker]

I am glad you had this issue ;) . After reading your process, it gives me an idea on how to do something very similar I hadn't tried to do yet.

[tret]

I am glad you had this issue ;) . After reading your process, it gives me an idea on how to do something very similar I hadn't tried to do yet.

Yeah? Do share your idea :grin:

[kricker]

It isn't much different than yours. I plan on using EvevntGhost to send a library update command just like you are doing. I just hadn't had time to investigate how to do it. But now I think I do ;). I will add one other step in between that will 1st organize and name the files appropriately.

[tret]

XBMC Atlantis Final
Ubuntu Intrepid 8.10


<advancedsettings>
<remoterepeat>1000</remoterepeat>
<musiclibrary>
<hideallitems>true</hideallitems>
</musiclibrary>
<videolibrary>
<cleanonupdate>true</cleanonupdate>
<hideallitems>true</hideallitems>
</videolibrary>
<video>
<usetimeseeking>true</usetimeseeking>
<smallstepbackseconds>4</smallstepbackseconds>
<smallstepbacktries>3</smallstepbacktries>
<smallstepbackdelay>300</smallstepbackdelay>
<timeseekforward>30</timeseekforward>
<timeseekbackward>-30</timeseekbackward>
<timeseekforwardbig>420</timeseekforwardbig>
<timeseekbackwardbig>-420</timeseekbackwardbig>
</video>
</advancedsettings>


This advancedsettings.xml file works perfectly with Ubuntu Hardy 8.04 and XBMC Atlantis Final, all settings here work within xbmc.

I took the same exact file and cleanonupdate is ignored in my new Ubuntu Intrepid 8.10 build w/ XBMC Atlantis Final. All other settings in this file work fine within xbmc.

Can anyone else w/ an Intrepid/Atlantis Final build duplicate this issue? The library is not cleaned when a library update takes place even though cleanonupdate=true in advancedsettings.xml.... again this worked without a hitch in Hardy.

Thanks,
Rob

[tret]

Bump!

So far I can only fix this by going back to hardy, weird

[althekiller]

Works fine here. Pastebin a debuglog repeating the problem.

[tret]

I have been able to reproduce the issue on both 8.04.1 and 8.10 and this time I am providing my log

1. x86 Ubuntu 8.04.1 (Fully up to date)
2. XBMC Atlantis Final
3. Video source is a permanently mounted cifs utf8 (using fstab) share from another linux computer (wired)
4. Standard Hardware, nothing fancy or not supported, Nvidia 9600GT (envyng used to install driver)
5. http://pastebin.com/fce3ece6

I first added a tv show section and set content, xbmc used thetvdb.com to get info on a couple of tv shows and the library worked fine. I manually deleted one of the shows from the source location and then ran an "update library" within xbmc, the library was cleaned with the update and the missing show was removed.

I then added a movie section and set content, XBMC used imdb.com to build the library and when the update was complete the movie and tv show library both worked fine. I again manually deleted one of the tv shows, ran an "update library" from within xbmc and this time the missing tv show was not removed from the tv show library. I attempted several more "update library" commands but the tv show was not removed. I then went and manually removed one of the movies and then ran an "update library" from within xbmc,the library was cleaned and the movie and the tv show were removed at the end of the update.

So far as I can tell it seems that when I have both a Movie and a TV show library, a removal of a Movie + an update will prompt a clean but a removal of a TV show + an update won't.

Any way to confirm this?

Thanks,
Rob

[xanadu1979]

So far as I can tell it seems that when I have both a Movie and a TV show library, a removal of a Movie + an update will prompt a clean but a removal of a TV show + an update won't.

Any way to confirm this?

Thanks,
Rob

I am experiencing the same thing on a fairly recent Windows build (about a week old from the SVN). I was going to post in the Windows forum but then I found this post.

I have both Movie and TV Show libraries. "cleanonupdate" works for movies, but my TV Shows aren't being removed when I delete them from the hard disk. But it does work when I manually trigger a Clean Library.

Not sure if it matters, but I'm using the MediaStream skin.

I'll post my debug logs when I get home.

**Edit**
Here is my debug log. I have the <cleanonupdate> setting at true in my advancedsettings file. The steps I used to make this log are as follows:

1. Set library to update on startup.
2. set <cleanonupdate> to true.
3. close XBMC
4. delete a TV show off my hard drive
5. start XBMC
6. notice that the tv show is still listed in my TV library
7. run a "Clean Library"
8. notice that the tv show is now gone.

Remember, it works fine for movies. So I know the <cleanonupdate> is doing something.

http://ubuntu.pastebin.com/f22be0f83

Almost forgot, the show I deleted was an episode of Oz.

[xanadu1979]

Edit

Nevermind. I was wrong on this post.

I am still having the issue from my last post though. Anyone else having problems with TV Shows not being cleaned with cleanonupdate turned on?

[tret]

Hi guys, I have posted on this issue before and didn't get anywhere so I am hoping that someone else here can confirm this issue for me.

Ubuntu 8.10
XBMC Atlantis or Latest SVN

I have enabled the cleanonupdate option in the advancedsettings.xml file. This works very well by the way in most ways but I have run across an issue w/ tv shows.



If I manually delete a movie file and run a video library update in xbmc, the missing movie is removed properly from the movie library

If I manually delete a movie file and a TV show or episode and run a video library update in xbmc, the missing movie and the missing TV show or episode are properly removed from the appropriate libraries

If I manually delete a tv show or episode and run a video library update in xbmc, the tv show or episode IS NOT removed from the tv show library

If I add a new tv show or episode and run a video library update in xbmc, the new tv show or episode is added to the tv show library, but the missing tv show or episode again IS NOT removed from the library

If I add a new movie and run a video library update in xbmc, the new movie is added AND the missing tv show or episode is removed properly



So it seems the only time cleanonupdate works for cleaning missing tv shows or tv show episodes is if something changes in the movie library. Can anyone else confirm this? Developers, is this by design? It would be nice if the cleanonupdate would work independently for the TV shows as well as for the Movies

tret

[sho]

This is now your third thread on the issue in 4 months.
I suspect either:
a) none else is using this option.
b) there is something specific to your installation.

If we could only hear from people using this with no problems.

[althekiller]

So did I just merge the fourth thread? Please stop making new ones it only makes a mess of the forums.

[tret]

sorry about the multiple posts, I'm pretty sure I only posted twice about this but regardless I'll stop. Is there anyone who cares to look into it at all? It does appear that at least 1 other user is having the same issue.

Can a few people test this out and see if you get the same results?

tret

[jmarshall]

Get it reproducible. This means:

1. A dummy filesystem, ideally as simple as possible that is used as a base for the scanning.
2. Full details of how to set up the system to reproduce (eg advancedsettings.xml you have setup).
3. Full details of the steps to go through to reproduce it.

In short: Make it easy for developers to reproduce a problem and it'll be fixed much more quickly.

Cheers,
Jonathan

[tret]

Get it reproducible. This means:

1. A dummy filesystem, ideally as simple as possible that is used as a base for the scanning.
2. Full details of how to set up the system to reproduce (eg advancedsettings.xml you have setup).
3. Full details of the steps to go through to reproduce it.

In short: Make it easy for developers to reproduce a problem and it'll be fixed much more quickly.

Cheers,
Jonathan

Thanks Jonathan, will work on this and get it posted. I appreciate your help

tret

[xanadu1979]

Get it reproducible. This means:

1. A dummy filesystem, ideally as simple as possible that is used as a base for the scanning.
2. Full details of how to set up the system to reproduce (eg advancedsettings.xml you have setup).
3. Full details of the steps to go through to reproduce it.

In short: Make it easy for developers to reproduce a problem and it'll be fixed much more quickly.

Cheers,
Jonathan

I posted my debug log earlier in this thread. The steps to reproduce are also there.

My TV shows are in Show Name => Season Number folders. They are on a physical hard drive in the htpc. They aren't on an external or network drive. My advancedsettings file only has the cleanonupdate tag. I'm using the Mediastream skin. I am deleting the files through the OS, not XBMC.

Again, cleanonupdate is working for my movies, but not my TV shows.

[jmarshall]

Unless you posted an actual file set, then that's not what I asked for at all (I probably didn't ask you for it either - lots of thread merges here ;)

Pretend we're the laziest people on the planet. Make it easy for us and we'll look into it. The absolutely worse thing for a developer to be doing is playing around trying (usually unsuccessfully) to reproduce something.

Cheers,
Jonathan

[tret]

Unless you posted an actual file set, then that's not what I asked for at all (I probably didn't ask you for it either - lots of thread merges here ;)

Pretend we're the laziest people on the planet. Make it easy for us and we'll look into it. The absolutely worse thing for a developer to be doing is playing around trying (usually unsuccessfully) to reproduce something.

Cheers,
Jonathan

Hi Jonathan,

I am totally willing to do what it takes to get you what you need to test this. Can you please let me know exactly what it is that you need. What do you mean by an actual file set? This issue is very easy to reproduce, I have been able to do it time and time again in all kinds of configurations. Let me know what you need and I'll get it done.

Thanks,
tret

[jmarshall]

I mean an exact replica of your filesystem, on which you can reproduce the same fault.

This is usually done by putting a bunch of 0 byte files, named identically to your movies and shows in a file structure identical to how you have things.

Simplify it down as much as possible (i.e. as few shows/movies as possible) so that it is still reproducible.

Then, zip it up, and post it on trac along with exactly how you scan (settings etc.), how you clean, and thus how you reproduce the problem.

Cheers,
Jonathan

[RockDawg]

I have <cleanonupdate> set to true in my advancedsettings.xml and movies no longer on my server are not removed from the video library. I have attached a debug log:

Apparently my log is too big for pastebin. Here's the first half:

http://pastebin.com/m44d9caaf

Here's the second half:

http://pastebin.com/ma3facda

Anyone experience seeing this too?

[RockDawg]

I updated to svn 17890, deleted my db and sources, created some dummy files, added their location as a source, and scanned them in to create a new db. I then deleted a one of the dummy movies and ran "update library" and the movie was not removed from the db. I have included a link which will download a .zip file I created that contains the folder with the dummy movies, the db that still contain the deleted movie (title is 'Unknown'), and a log taken right after I ran the "update library". Also, here's the contents of my advancedsettings.xml:

<advancedsettings>
<loglevel>3</loglevel>
<videolibrary>
<cleanonupdate>true</cleanonupdate>
</videolibrary>
</advancedsettings>

Here's the .zip file:

http://www.box.net/shared/6kj6qd5cm2

[kricker]

I see the same thing and have for some time.

[RockDawg]

Any word on this?

[jmarshall]

Add a ticket on trac (if there isn't one already) and add your files to that.

Someone will get to it when they have time.

[ouija]

Grrr.. my system is not cleaning the library on update either...

[philter]

My system is also not respecting the cleanonupdate setting.


edited to add it doesn't clean on update with TV shows...as others have reported it does work with movies

[philter]

Did someone add a trac ticket? Any update?

[xanadu1979]

Still isn't working as of the SVN from a few days ago. I don't think the trac ticket ever got added.

RockDawg's situation doesn't seem to be the same as some of the rest of us. My movies are getting cleaned just fine, it's my TV Shows that aren't getting cleaned.

philter and I seem to be having the same issue.

I'll try to get some logs and a dummy filesystem together tomorrow. I'd really love to see this get fixed.

[xanadu1979]

I don't think I'm going to have time to get a dummy filesystem together. Hopefully someone else can take the initiative and get it done.

But really, I think there's more than enough info in this thread to look into it. A few of us have posted logs, I detailed the exact steps to reproduce. I don't see why a dummy filesystem is needed because it happens with every file on a pretty standard setup.

Here are the exact steps to reproduce, quoted for an earlier post in this thread that I made a while back.

I think it's important to note that it seems to happen when you have both movies and TV shows in the library. You need both.

Also, my TV shows are stored on a local NTFS partition. My user has read/write access to the files.


Here is my debug log. I have the <cleanonupdate> setting at true in my advancedsettings file. The steps I used to make this log are as follows:

1. Set library to update on startup.
2. set <cleanonupdate> to true.
3. close XBMC
4. delete a TV show off my hard drive
5. start XBMC
6. notice that the tv show is still listed in my TV library
7. run a "Clean Library"
8. notice that the tv show is now gone.

Remember, it works fine for movies. So I know the <cleanonupdate> is doing something.

http://ubuntu.pastebin.com/f22be0f83

Almost forgot, the show I deleted was an episode of Oz.

[hamse]

I still haven one problem. It works fine for the movie section, but I can't get it to remove tv episodes...

[xanadu1979]

I still haven one problem. It works fine for the movie section, but I can't get it to remove tv episodes...

This is sort of a known problem. I say 'sort of' because there is a thread in the Linux forum about it but nobody seems to want to take the initiative to look into fixing it.

I and others have posted in that thread. The developers have asked for more information. I feel that we have provided plenty.

Here is the thread:
http://xbmc.org/forum/showthread.php?t=41111

Feel free to bump it up with information (debug logs) detailing your issue. If you wish to get together the dummy filesystem, I would greatly appreciate it. I don't have time to do it. But like I said in my last post, I don't see why it's necessary.

Sadly, it's probably too late to get it fixed for Babylon. Hopefully it can be fixed for the next release.

[tslayer]

Obviously it is not high on the priority list.

But patches are more than welcome for quicker resolution/fix.

[tslayer]

Oh, and further to this, I don't think anybody ever created a trac ticket (maybe I am mistaken). Without one, it won't be looked at at all.

[hamse]

thanks for the help... I will play around a little...

[spiff]

this doesn't work for eps? that is surely a bug

[xanadu1979]

this doesn't work for eps? that is surely a bug

It seems to work fine if a movie needs to be cleaned in addition to a tv episode.

It's when no movies are cleaned that it seems to skip cleaning the tv episodes.

It's hard to pin down and explain (probably why nobody has looked at it). In the most simple terms:

1. set cleanonupdate to true in advancedsettings
2. set the option that does a library update when xbmc starts to true
3. don't have xbmc running, delete a tv show off your hard drive
4. don't touch your movies, don't delete one or add one
5. start xbmc
6. when the library update is done, the tv episode you deleted is probably still in your library.

That should reproduce it. For me, it only happens when running the update on xbmc startup. Manually triggering an update works fine (if I remember correctly).

[sho]

Where is the ticket?

[spiff]

thanks xanadu should allow me to reproduce

[xanadu1979]

Where is the ticket?

Just created one:
http://xbmc.org/trac/ticket/6493

[sho]

Thanks :)

[sho]

Trac tickets and grumpy ones do wonders:
http://xbmc.org/trac/changeset/19930

[kricker]

Woot! That grumpy bastard sure can be helpful.

[RockDawg]

It's still not working for me.

Log: http://pastebin.com/m385a1390

19:45:04 T:4556 M:1482973184 ERROR: DIRECTORY::CDirectory::GetDirectory - Error getting smb://TOWER1/Movies/The Wrestler/
19:45:04 T:4556 M:1482969088 DEBUG: Thread 4556 terminating
19:45:04 T:1288 M:1480953856 ERROR: DIRECTORY::CDirectory::GetDirectory - Error getting smb://TOWER1/Movies/Youngblood/
19:45:04 T:1288 M:1480949760 DEBUG: Thread 1288 terminating

Those two movies are the ones that are gone from the server and should have been cleaned.

[RockDawg]

Can someone please confirm this so the devs might investigate further. I've tried my existing db, I nuked it and started over, and I've done this on two different machines and clean on update doesn't work in any scenario. I can't be the only one can i?

[sho]

I think your best bet to get attention from the devs would be to comment on the tracker

[wildonrio]

Can someone please confirm this so the devs might investigate further. I've tried my existing db, I nuked it and started over, and I've done this on two different machines and clean on update doesn't work in any scenario. I can't be the only one can i?

You're not the only one. It's not working for me either.

[spiff]

i need exact reproducing instructions. i do not have time to both find the bugs and fix them

[toiva]

Well in my case, my movies are in subfolders with "use folder nanes for lookups enabled". If i delete a movie folder and then in movie library do library update, it won't cleanup.
Tv shows seem to clean up properly.

http://koti.kainuu.com/manki/xbmc.log

Titanic (:o) is a nonexistent folder in that log.

[RockDawg]

Well in my case, my movies are in subfolders with "use folder nanes for lookups enabled". If i delete a movie folder and then in movie library do library update, it won't cleanup.
Tv shows seem to clean up properly.

http://koti.kainuu.com/manki/xbmc.log

Titanic (:o) is a nonexistent folder in that log.

Thanks for the confirmation toiva. I was going to update the ticket on the tracker, but I got busy plus I wanted to do more testing to be sure of the problem re-opening a ticket. I get the exact same as toiva. TV Shows clean fine, but movies don't. I also have my movies stored in individual folders. I just tried nuking the db and I created some dummy files to see if it's related to folders or if indeed individual files have the same problem. After scanning the dummy files and then removing one, update library does not remove the missing movie. I have attached a log that is with the dummy files since you already have a couple with movies in folders.

Log: http://pastebin.com/m38852b86

I'm certainly no expert, but this line would seem to indicate that it knows the contents have changed, but it still doesn't do anything for some reason:

[CODE]Hash[movies,smb://KEVIN-DESKTOP/Users/Kevin/Videos/]:DB=[141C82A9E6423872CDD40ED5B46BB9AC],Computed=[6B12344BB4E29E6DADC6637D585B8EEE]/CODE]

[spiff]

r20289. cheers.

[RockDawg]

r20289. cheers.

No, cheers to you for the fix! I'll be sure to verify as soon as there's a Win build available that includes this revision. Thanks again!

[kricker]

No, cheers to you for the fix! I'll be sure to verify as soon as there's a Win build available that includes this revision. Thanks again!Okay, new build on the way.

[RockDawg]

Okay, new build on the way.

Cool!

[kricker]

Cool!
Posted (http://xbmc.org/forum/showpost.php?p=336320&postcount=570).

[RockDawg]

It seems to be fixed for single file movies, but not foldered movies. I've tried both. Here is a log after I updated the library consisting of foldered movies: http://pastebin.com/m6c20b534

The missing movie in my case is Youngblood and as you can see, it shows an error finding it, but it is not removed from the library.

16:50:33 T:2360 M:983691264 ERROR: DIRECTORY::CDirectory::GetDirectory - Error getting smb://TOWER1/Movies/Youngblood/

P.S. Thanks for the quick build kricker even though it took me a while to get back on this.

[RockDawg]

Bumping in case this got lost in the shuffle.

[RockDawg]

I've done some more testing on this and I have noticed something. It only does it with movies that have the standard DVD file structure. I've tried single file movies (.avi, .mpg, .wmv, .mkv) and they update correctly. Here's a relevant log where the update removed the missing movie:

http://pastebin.com/m2776ad0d

Also, I then put those same single file movies within individual folders and enabled "Use folder names for lookups" on the initial scan. After deleting one the the folders (and it's accompanying movie file), running update library did indeed remove the missing movie from the db. Here is a log that shows that:

http://pastebin.com/m745b1b05

But whenever I make it so that the movies are folders containing VIDEO_TS.ifo, updates refuse to clean the library of missing movies.

[feerlessleadr]

I ran into a weird problem just now when trying to clean up my library.

I am trying to remove some shows from my TV Shows Library and for some reason, everytime XBMC updates the library it adds them back in.

I wanted to get rid of these shows, so I figured the easiest way to accomplish this was to delete the shows folder from the TV folder on my computer, and then when the XBMC library auto-updates on startup, they would automatically be removed. (and yes I do have the clean on update string in my advanced settings)

<advancedsettings>
<videolibrary>
<cleanonupdate>true</cleanonupdate>
</videolibrary>
</advancedsettings>

After I deleted the shows and started XBMC and let it update the library, I saw that the 3 shows were still present. Thinking this was strange, I just manually removed them from my library without a second thought. I closed XBMC and started it back up a little while later, and noticed that during the auto update, it downloaded the information for the 3 shows that I just removed. perplexed, I looked in my TV shows folder on my computer and the files were still deleted, and not in the recycle bin either. I removed them again, and bam, they again get added when I update the library.

I checked to make sure that there weren't any hidden folders that I didn't know about in my TV show directory, and there are none.

Please also note that after the shows are re-added to the library, there are no episodes listed, just the show itself.

I have no idea why this is happening, anyway, here is a debug log.

http://pastebin.com/m6f54de2

Any help will be greatly appreciated.

Also, my directory structure is as follows:

G:\TV Shows\**TV SHOW**\Season **\**filename.avi/.mkv**

The above 3 listings are the shows that won't get deleted

::UPDATE:: I just tried the clean library option in the video settings and it got rid of the 3 TV shows, so I guess XBMC is not cleaning my library when I update it, as specified in the advanced settings. The advanced settings should be in the "Roaming" userdata folder correct, and not the "Program files" userdata folder?

[Hitcher]

I thought this function was disabled to stop remote shares from being removed if they weren't accessible at startup.

[feerlessleadr]

I thought that is why it was moved to the advanced settings and disabled by default.

[RockDawg]

I created a ticket for this: http://xbmc.org/trac/ticket/6629

I created a new ticket rather than re-open the other because it was specifically concerning TV shows and this concerns movies. I didn't want to make a faux pas.

[sho]

I am going to append this to the cleanonupdate isse thread.
There seem to some outstanding issues still.

[RockDawg]

Any word on this?

[kulprit]

I'm actually still having the problem with TV shows on a recent build, will find out some more info and come back.

[RockDawg]

It's been quite a while. Is there any chance this gets fixed sometime soon? My DVD file structure videos (vobs, ifos, bups) still do not get cleaned. I used to love this feature of XBMC, but it has been broken for so long I wonder if there are any plans to fix it.

[RockDawg]

Also to reiterate, running 'Clean library" from settings/video removes the missing movies as expected. Why does update library act different?

[Hitcher]

Also to reiterate, running 'Clean library" from settings/video removes the missing movies as expected. Why does update library act different?
Because you don't want disconnected shares being wiped from the DB every time you update.