As a follow up to my last post, scripters and plugin writers should be aware of the best practises for accessing files under python:

1.  Don’t use the old Q:, P:, T:, U:, Z: paths. Replace these with their special:// counterparts below.

2.  Use xbmc.translatePath() to translate any path to it’s true path if/when you need to read or write from the filesystem.

Note that xbmc.translatePath() will take care of number 1 for you in the meantime, but this functionality is not guaranteed to be available forever.

So what should you use?

1.  Use special://home/ if you need to read or write to the writeable version of the xbmc install folder.

2.  Use special://profile/ if you need to read or write to the current users userdata folder.

3.  Use special://temp/ if you need to read or write to a temporary location.

If you absolutely have to, you can also:

4.  Use special://xbmc/ to read from the xbmc install folder itself.

5.  Use special://masterprofile/ to read from the master users’ userdata folder.

These should not be needed by the majority of scripts, however, so if you find yourself needing to do either of 4 or 5, please contact me so that I know why, and can see whether there’s a better way to do things.

Hi – long time since my last blog – I’ve been too busy with real life stuff, not to mention XBMC related activities!  This post is to detail the rather boring, but very important path cleanup work that WiSo and I have just completed.

As a remnant from the xbox days, XBMC used 5 internal DOS based drives as base paths in order to find files and store settings and so on.  These were Q: (the location of the xbmc executeable), Z: (a temporary cache), P: (the current profiles userdata folder), U: (a writeable version of Q:), and T: (the master users userdata folder).  References to these littered the code, and caused many a headache with the initial port to Linux, where essentially it was decided to wrap the win32 filesystem layer that we had in place, and, in addition, provide a path translation routine (CUtil::TranslatePath, macro’d to _P()) to translate from the virtual path to the real path.

On win32 we didn’t have to bother with this, but we did have to (internally) map those drives to the appropriate directories.  This, ofcourse, caused problems for anyone who wanted to use those drive letters for removable storage or network drives. Read more…

Just a quick post of gratitude for all the bug reports we’ve had over the last month or so.  Without your great support at testing things out and taking the time to narrow down and get any issues reproducible, XBMC wouldn’t be as great as it is now.

As you can see, we’ve been busy tackling these reports – 228 closed tickets out of 326 at the time of writing, so we’re 70% of the way there, and that includes only bugs that have actually had tickets assigned (i.e. not stuff we’ve discovered ourselves or was reported on the forums that there was an easy fix for that didn’t really warrant ticket creation).  sho has been doing an awesome job at managing the trac reports – busy assigning them to us devs and following up on them to make sure that they get fixed.

We hope to have the first beta release out within a week, and will be shortly inviting some of our more active users on the forum and trac to come aboard the team as official beta testers.  There’ll be some nice surprises included with the beta as well that we hope you guys will smile about, including a treat or two for you Mac users.

Let’s keep up the testing and bug bashing to make Atlantis as stable as possible.  Your help in looking over the trac reports and reproducing and supplying more information about the reports makes our job much, much easier, so keep it up!

I was testing some smartplaylists out from some bug reports, and found that the same query was performed significantly faster in SQLiteSpy than in XBMC on the same machine.  The performance was an order of magnitude slower in XBMC than it was in SQLiteSpy, and we actually use a newer version of SQLite than is in the version of SQLiteSpy I was testing.

Turns out that we’re using a “wrapper” around the lib, which uses sqlite3_exec() and a callback routine to grab the per-row data.  This isn’t too much of a problem (see below) and is certainly an easy way to go about things.  The slow downs were caused by the way the callback was implemented.

The main issue was due to allocation + deep copying. For each row of data, the callback gets called, and we copied the data into a row_type = std::map<int, field_value>, where field_value was basically a string.  The row is then pushed into a std::map<int, row_type>.  This, ofcourse, results in numerous allocation + deallocation as the maps are copied + moved around in memory, in order to optimize the map lookup.  The use of map here is the first obvious problem: It’s indexed by the row number, so why not use a vector?  Furthermore, if we use a vector<row_type *> then it’ll save constantly copying data whenever we have to resize the vector.

The second problem isn’t quite as bad, but is inherent to the way sqlite3_exec() is handled.  Each field in each row is converted to a string to be passed to the callback, and we must then go and convert it back to the appropriate type for actual use.  We can skip this step by breaking the sqlite3_exec() down into what it does behind the scenes (sqlite_prepare/step/column/finalize) and just copy the appropriate datatype straight away, ensuring that no conversion needs to be performed.

Result: The old version took 1760+19862 ms (query + retrieval), whereas the new version takes just 324+1370 ms.  Not bad for an hours effort.

Discovered a nasty feature of win32’s feof function.  We have our own dll loader on win32 that automagically handles linking up any of the usual filesystem functions (fopen, fclose et. al.) into our virtual filesystem versions.  This allows dll’s to not have to care about where they’re getting their files from, which is neat.  Found a problem the other day where a user was reporting lockups when XBMC attempted to load zero byte sized jpg files.  This seemed rather strange to me, and I initially suspected problems in the image library we use (cximage).  Debugging dll’s is never a fun experience when you aren’t loading them using the normal win32 LoadLibrary functions – basically one is reduced to printf.  After an hour or so of running through various massive log outputs, I figured out that the problem was an infinite loop in the implementation of the BMP loading function.  It basically ran a loop reading in characters from the file via getc and checking for EOF using feof.  The weird bit was that our exported version of feof was not being called.  How could this be?  More investigation showed that the dll wasn’t actually including it as a wanted import at all – somehow it had it’s own implementation of feof built in.

The reason was this: win32 defines feof using both a macro and a function. The macro was the version it was calling, and obviously wasn’t working, as the FILE * object it was operating on was the structure that we use to store everything we need to know about the virtual filesystem layer, and was thus not returning EOF.

A quick #undef feof later, and we were back in business.

Further to my previous discussion, I’ve been playing with ways to improve navigation.  There’s two things that have just hit SVN:

Jump by letter down a list

Two new actions allow you to jump through a list by the first letter of the items (in terms of their sort order).  This probably needs a little more tweaking for certain usage scenarios but it works well for alphabetically sorted lists.  For numerically sorted lists (eg sort by track number) it’s not so useful, but then again sorting by track number isn’t useful for long lists anyway.  One idea I have is to ensure that the number of unique “sort letters” is sufficiently high (eg at least 10) before enabling it in a list – for instance, if we have 100 strings all starting with the letter A, we could switch to the first 2 characters as the identified piece.

Highlight the first letter while scrolling

This is pretty cool – as you scroll quickly, we have a new visibility condition Container.Scrolling which comes on, allowing the skinner to fade something in and out based on scrolling.  This, combined with the new ListItem.SortLetter info label allows the skinner to have a large letter pop up when you’re scrolling with the first letter of the sort order.  Quite a nice addition (even with my rather shabby skinning).

Currently I’m working on refining the “live searching”, based on some of the work done by Bripeace for live filtering using the remote (and now keyboard).

My current thoughts are that search is more useful in general than filtering – filtering only has use for the current list, which is really just another method for scrolling through the list.  If you are looking for something in particular, then searching is the way to go.

Search in the music library is quite nice already, albeit the input method is a bit clunky.  There’s no edit control, so all users (no matter the input device) have to go through the virtual keyboard to enter the search string.  The search is then performed in the background (unthreaded though currently) by hitting the db and updating the list below.  If we can get rid of the big virtual keyboard that’s effectively hiding the list (i.e. use an edit control for keyboard + remote with SMS input users, and ideally a smaller keyboard for gamepad users) then I think this will be really nice.

Assuming search is done really nicely, then filtering could be reduced to just a method for faster jumping through the list – i.e. don’t actually remove items from the list, but instead jump through it.  There’s some issues with this though – the main one being that lists need not be alphabetically ordered, so this needs some thought.