XBMC Community Forum  

Go Back   XBMC Community Forum > Development > Scraper Development

Scraper Development Developers forum for meta data scrapers. Scraper developers only!
Not for posting feature requests, bugs, or end-user support requests!

Reply
 
Thread Tools Search this Thread Display Modes
Old 2007-06-16, 19:29   #1
ezd
Member
 
Join Date: Oct 2003
Posts: 39
ezd is on a distinguished road
Default Scraping inconsistency scrap.exe/xbmc?

I'm making a scraper for AsianDB.com. It seems to work flawlessly under scrap.exe, but XBMC misses a lot of info it retrieves. Here's an example details XML output:

Code:
<details>
	<title>Violent Cop</title>
	<year>1989</year>
	<director>Takeshi Kitano</director>
	<runtime>103mins</runtime>
	<thumb>http://www.asiandb.com/data/title/mini/4141.jpg</thumb>
	<rating>7</rating>
	<votes>3</votes>
	<genre>Action</genre>
	<genre>Crime</genre>
	<credits>Takeshi Kitano</credits>
	<credits>Hisashi Nozawa</credits>
	<actor>
		<name>Takeshi Kitano</name>
	</actor>
</details>
XBMC doesn't extract the director, genre, credits (correct way to enter writers?) and actors, but does get all other items.

Is there a bug in my XML output? (Note: pretty-printed for readability, no extra whitespace in actual XML)

Also, pressing X+Y during boot did get me in debug mode, but didn't tell much about the scraping process. Is there a method (like in the old days ) to set the debuglevel to 'insane' or similar?

Thanks for any help you can give,

ezd
ezd is offline   Reply With Quote
Old 2007-06-16, 20:24   #2
ezd
Member
 
Join Date: Oct 2003
Posts: 39
ezd is on a distinguished road
Default

For reference, I've upped the current asiandb.xml to pastebin.
ezd is offline   Reply With Quote
Old 2007-06-18, 08:41   #3
spiff
Grumpy Bastard Developer
 
spiff's Avatar
 
Join Date: Nov 2003
Posts: 7,715
spiff is on a distinguished road
Default

i'm on a conference this week, so this post is only to say that i cannot see anything wrong at first glimse. i hardly have inet accessiblity so i have to wait until i get back home to investigate.
__________________
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
spiff is offline   Reply With Quote
Old 2007-06-18, 19:57   #4
ezd
Member
 
Join Date: Oct 2003
Posts: 39
ezd is on a distinguished road
Default

Thanks for the heads up, no hurry here, mostly did this for the Greater Xbmc Good

Enjoy your conference!
ezd is offline   Reply With Quote
Old 2007-06-26, 16:40   #5
spiff
Grumpy Bastard Developer
 
spiff's Avatar
 
Join Date: Nov 2003
Posts: 7,715
spiff is on a distinguished road
Default

before each of those you have regexp's that grabs the relevant pieces of the html. on those you don't specify 'noclean="1"' and hence all html tags are stripped off. i guess the scrap.exe doesnt honor this.
__________________
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
spiff is offline   Reply With Quote
Old 2007-07-08, 15:13   #6
blaize
Member
 
Join Date: Oct 2006
Posts: 63
blaize is on a distinguished road
Default

any progress on this scraper ?
i really need this one.
blaize is offline   Reply With Quote
Old 2007-07-09, 10:56   #7
spiff
Grumpy Bastard Developer
 
spiff's Avatar
 
Join Date: Nov 2003
Posts: 7,715
spiff is on a distinguished road
Default

then i suggest you finish it
__________________
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
spiff is offline   Reply With Quote
Old 2007-07-09, 11:34   #8
blaize
Member
 
Join Date: Oct 2006
Posts: 63
blaize is on a distinguished road
Default

wise-ass... if i could don't you think i would ?
some people have learned them selfs programming skills, other artistic skills.
blaize is offline   Reply With Quote
Old 2007-07-09, 11:38   #9
spiff
Grumpy Bastard Developer
 
spiff's Avatar
 
Join Date: Nov 2003
Posts: 7,715
spiff is on a distinguished road
Default

it doesnt take programming skills. that's the whole reason i created the scraper system. it only takes some logic and reading a 10 min regexp guide.
__________________
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.
spiff is offline   Reply With Quote
Old 2007-07-09, 11:57   #10
blaize
Member
 
Join Date: Oct 2006
Posts: 63
blaize is on a distinguished road
Default

if you think it's that easy for everyone, then why is 'esd' having problems with it ?
I'm pretty much code-blind, but if you (or anyone else) could give a little help i might give it (another) try.
blaize is offline   Reply With Quote
Reply

Bookmarks


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 18:21.


Protected by Akismet, We recommend WordPress blogs
Copyright © 2008, XBMC Project