moviemaze.de scraper development - help needed

[w00dst0ck]

Hi there,

im currently developing a http://www.moviemaze.de scraper. I managed to scrap most of the provided informations but have some trouble with the following issues:

Director:

Moviemaze.de HTML

									<td valign=top class="standard">
											<span class="fett">Regie:</span>
										</td>

										<td valign=top class="standard_justify">
										Guillermo Del Toro										</td>
									</tr>

my regex:

<RegExp input="$$6" output="&lt;director&gt;\1&lt;/director&gt;" dest="5+">
	<RegExp input="$$1" output="\2" dest="6">
	<expression trim=2">Regie([^&quot;]*)&quot;standard_justify&quot;(.*?)&lt;</expression>
	</RegExp>
	<expression>([A-Za-z0-9 ,.]+)</expression>
</RegExp>

scap.exe delivers the right results, but XBMC displays a TRIM as result.
I decided to get the result in two steps because it's surrounded by tabs.
Any ideas why XBMC displays TRIM?


Actors:

Moviemaze.de HTML

									<tr>
										<td valign=top class="standard">
											<span class="fett">Darsteller:</span>

										</td>
										<td valign=top class="standard_justify" width=100%>
										<a href="/celebs/89/1.html">Christian Bale</a> (Bruce Wayne/ Batman), <a href="/celebs/114/1.html">Heath Ledger</a> (Joker), <a href="/celebs/127/1.html">Michael Caine</a> (Alfred), Maggie Gyllenhaal (Rachel Dawes), Gary Oldman (Lt. James Gordon), Aaron Eckhart (Harvey Dent), <a href="/celebs/47/1.html">Morgan Freeman</a> (Lucius Fox), Monique Curnen (Detective Ramirez), Ron Dean (Detective Wuertz), Cillian Murphy (Dr. Jonathan Crane)										</td>

									</tr>

My regex:

<RegExp input="$$2" output="&lt;actor&gt;&lt;name&gt;\1&lt;/name&gt;&lt;role&gt;\2&lt;/role&gt;&lt;/actor&gt;" dest="5+">
	<RegExp input="$$1" output="\2" dest="2">
		<expression repeat="yes">Darsteller:([^%]*)%&gt;(.*?)&lt;/tr</expression>
	</RegExp>
	<expression repeat="yes" trim="1">([A-Za-z \-]*)\(([A-Za-z \-]*)\)</expression>
</RegExp>

It works, but I miss the actors with a href link and I did not managed to find a solution.


German Umlaut [äöüß]:

I can't grep words with characters like "german umlaute" / [äöüß].
I've tried expressions like ([A-Za-z0-9äÄöÖüÜß ,.]+) and ([A-Za-z0-9äÄöÖüÜ ,.]+) without success.


Can somebody please help me?

regards,
w00dst0ck

[spiff]

if that's a c&p;

<expression trim=2">Regie([^&quot;]*)&quot;standard_justify&quot;(.*?)&lt;</expression>

missing a "

actors; try adding the href stuff with a ?, i.e. make it optional

umlaut might have to do with encoding - make sure the scraper is set to the encoding you are saving the file in (and also that it matches moviemaze's encoding)

[w00dst0ck]

thanx for reply!

Originally Posted by spiff

if that's a c&p;

<expression trim=2">Regie([^&quot;]*)&quot;standard_justify&quot;(.*?)&lt;</expression>

missing a "

Added the missing " but there is no difference. Also tried without the trim option. But that results in no information provided.

Originally Posted by spiff

umlaut might have to do with encoding - make sure the scraper is set to the encoding you are saving the file in (and also that it matches moviemaze's encoding)

How do i set the encoding of the scraper?
The moviemaze.de page is encoded with iso-8859-1 and my results are generated with:
<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>

[spiff]

you set the encoding of the scraper xml file using exactly that kinda header as you just pasted

[Gaarv]

Originally Posted by w00dst0ck

Added the missing " but there is no difference. Also tried without the trim option. But that results in no information provided.

Seems you're missing a ">" too after "standard_justify". Its not required to match, but it will be returned in /2 so thats a problem.

For umlaut, like spiff already said use ""so-8859-1" but be sure if you're using wilcard matches to include "&", "#" ";" and numbers because umlaut is displayed as it in the source :

Heerf & # 2 5 2 ; hrer
Ma & # 2 2 3 ; arbeit

Heerführer
Maßarbeit

I had to space out the code to illustrate, of course they aren't needed.

[w00dst0ck]

Originally Posted by Gaarv

Seems you're missing a ">" too after "standard_justify". Its not required to match, but it will be returned in /2 so thats a problem.

Thanx, that solves the problem.

The problem with the umlaut is solved by using (.*)

Next I will try to solve my href problem...

[Gaarv]

I did this rapidly, you will have to adapt ot limit its application in the whole page but it gave the idea

[^=",]*([A-Za-z0-9 ./]*\([A-Za-z0-9 ./]*\))

In the exemple you gave this match all actor names and role, with sometimes a </a> that needs to be cleaned

[w00dst0ck]

Hi there and thanx for your help.
I've only one problem left with the actors part.

									<tr>
										<td valign=top class="standard">
											<span class="fett">Darsteller:</span>

										</td>
										<td valign=top class="standard_justify" width=100%>
										Til Schweiger (Ludo), Nora Tschirner (Anna), Matthias Schweighöfer (Moritz), Alwara Höfels (Miriam), Jürgen Vogel (Jürgen Vogel), Rick Kavanian (Chefredakteur), Armin Rohde (Bello), Wolfgang Stumph, Barbara Rudnik (Lilli), Christian Tramitz										</td>
									</tr>

<!--Actors-->	
	<RegExp input="$$2" output="&lt;actor&gt;&lt;name&gt;\2&lt;/name&gt;&lt;role&gt;\5&lt;/role&gt;&lt;/actor&gt;" dest="5+">
		<RegExp input="$$1" output="\2" dest="2">
			<expression trim="2" repeat="yes">Darsteller:([^%]*)%&gt;(.*?)&lt;/tr</expression>
		</RegExp>
		<expression repeat="yes">(&lt;a href\="[^&gt;]*&gt;)?(.*?)(&lt;/a&gt;)?( \((.*?)\))?, </expression>
	</RegExp>

The 10 tabs in front of the first actor name are added in \2. I've tried to clear them out with [\t]{10}?
but i think the XBMC scraper engine don't understand \t.
Any idea how I can get solve this?

[spiff]

\\t

[w00dst0ck]

Found another solution.

Will submit the working scraper at http://xbmc.org/trac/ticket/4563