Getting started with scraper development? Discoogs, and Anime News Network

[ztripez]

Hello there fellow xbmc:ers,
I've started a small project of mine writing 2 scappers for;

Discoogs (Music), one of the most complete database for electro-music, something the allmusic scrapper lacks.
http://www.discogs.com/

and
Anime News Network (Tv), have a very big database of anime with episode names etc.
http://www.animenewsnetwork.com/

Nevermind, to the point;
I have to restart xbmc everytime, takes forever just to make a small change in the scapper-xml. I've read on the forum about some small program called scap.exe but i can't find it anywhere. Perhaps it is in the source but have no posibility to compile the xbmc here (various reasons).

So how do you do it?

[Trazer]

I was just told (by spiff) in the moviemeter scraper topic that scrap.exe is outdated and as a result can not be used to effectively test a scraper.

[ztripez]

Oh, so there is no other way to do then the "hard" way.. well well not much to do.

[spiff]

why do you have to restart xbmc?

and obviously using the linux/osx/windows versions would be easier than xbox

[ztripez]

well perhaps i have a bad build or something but if i don't restart xbmc the scrapper changes don't get update properly.
Yes I'm writing the scraper on a windows build atn (will move it to my xbox when it's done).

[spiff]

ah, darn, did i forget to fix the reload on refresh..

if you can build, here's a quick'n'dirty hack that might make your life a little bit easier.

GUIWindowVideobase.cpp, comment line 633 m_database.SetDetailsForMovie(item->m_strPath, movieDetails);

that way info won't be stored in the db. so you can do the lookup, close the dialog, do the lookup etc and scraper should reload

[ztripez]

hm.. i seems to me that i will ditch the windows build and use my ubuntu comp instead for scraper develop, since i can't build on Windows.

I'll try the small hack you gave when i got a good build enviroment up and running on Ubuntu.

When I'm at it, is there any info about how the music scrapers works? I'm just starting to get the hang of how the buffers and stuffs work .

Thanks for the help.

[spiff]

allmusic.xml is the documentation

basically it's just like movies, except that we have two search functions one for albums, one for artists

documenting stuff is not one of my "strong" sides

[ztripez]

Yeah i'm trying to reverse-engineer the allmusic.xml but i think i'm not getting hang of some fundmental things.

From the begining:
The discoog search string for releases (album) look like this:

http://www.discogs.com/search?type=releases&q=[Album]&btn=Search

if I understand correctly the xml-code would look something like this:

	<CreateAlbumSearchUrl dest="3">
        <RegExp input="$$1" output="http://www.discogs.com/search?type=releases&q=\1&btn=Search" dest="3">
            <expression noclean="1"></expression>
        </RegExp>
	</CreateAlbumSearchUrl>

Where $$1 is the Album name and the nonclean=1 means that $$1 will not be empty after the search, correct?

the result of the search will be stored in $$3, correct?

Now for make things abit more complicated, i know that you will get more narrowed searched (discogs db is huge) if you include the artist name in the search. So a better search url would be
http://www.discogs.com/search?type=releases&q=[Artist][Album]&btn=Search
any tips on how i do this?


Yeha i know how it is with the documentation-part, works as a developer myself ;P

[ztripez]

nevermind about the noclean=1, found info about it on the wiki.