OFDB scraper

[morte0815]

Hi,

This is my first "release" of the ofdb (germen version of imdb) scraper.

The mainfeatures work, but there are some problems:

- Umlauts are not readable (??siteencoding??)

- i parse the genres into individual tags (not only one genre tag; mabe one could change this in the scraper parser code (In databases you should not store lists in attributes )) If this will not be changed, then i will change the parser...

- atm no original title since there is no tag for it
- mpaa is only fetched if there is the movie was in the cinemas... (addition here: maybe some possibility to check if a regex failed. example:

PHP Code:

<regex name"theregextocheck" .../>
<Regex name="aName" condition="theregextocheck"> <!--this one will only be called if the condition does match--> 


cheers morte

[spiff]

i have no idea how ofdb expects its urls to be encoded. it certainly does not use normal url encoding (umlauts SHOULD be encoded, it does not accept that).

several <genre> tags, sure. i dont see why this is easier to do than the current / separated list though. as for storing them like that in the db, its to speed up the queries, constructing the / separated string each time takes time....

just add the original title as a tag, we will get it in there eventually.

as for those conditions, they are easily simulated using two regexps + clear="#ofbuffer". let one regexp grab the conditional block, clearing the buffer no matter what. then the next expression will either have nothing to look through (and hence fail), or you did grab something during the first expression so they will succeed....

[morte0815]

encoding the urls is the one thing... but the other thing is, that you cant read the results. in the list, where you can choose the the movie the umlauts work like a charm, but e.g. the plot is not readable if there are umlauts in it.

the thing with the genre is given through a db design pattern... it is normal not to store lists in an attribute. it also should make the genre-queries easier, since you do not have to split the result of the genre but only to iterate through all genre tags to find the movies with the needed genre...

the original title will be added, and the condition thing will be testet

thanks for the response

morte

[sCAPe]

spiff: Any chance of integrating this ofdb scraper to SVN when its stable and finished? Or should the users download this scraper by themselves?

Don't know how it is planned for all future scrapers?

sCAPe

[spiff]

yes ofc we will stick it in svn.

everything deemed to be of high enough quality will be stuck in svn. the more the merrier

[morte0815]

So here is the version with the original title (<originaltitle>)!

It also reorders the title: Matrix, the => The Matrix
If someone does not want this behaviour, then tell me and i write another version..

CYA Morte

[asg]

<RegExp input="$$1" output="&lt;horst&gt;\1&lt;/horst&gt;" dest="8">

.. horst

Thanks
asciii

[morte0815]

well, thats some debugging tag... forgot to delete it.. but as long as it works...

AND NO: MY NAME IS NOT HORST!

cheers morte

[Solo0815]

THX!
I'll try it later this evening and report back!

BTW: Horst is a nice name for a debugging tag Are there Karl and Hans in the script, too *LOL* jk

[Solo0815]

encoding the urls is the one thing... but the other thing is, that you cant read the results. in the list, where you can choose the the movie the umlauts work like a charm, but e.g. the plot is not readable if there are umlauts in it.

Thats the only "bug" i noticed so far. Nice scraper!