View Full Version : Does XBMC and XBMC Media Center end-users have permission to scrape imdb.com website?
ultrabrutal
2009-01-12, 10:12
Just to take one, imdb:
IMDB Conditions (http://www.imdb.com/help/show_article?conditions)
"Robots and Screen Scraping: You may not use data mining, robots, screen scraping, or similar data gathering and extraction tools on this site, except with our express written consent as noted below."
Do XBMC have this permission? Just curious...
no, i was not aware of those terms (or atleast i could claim so).
not so much any longer. r17032 disables the scraper
ultrabrutal
2009-01-12, 14:47
Shit! What a party pooper I turned out be :(
I'm afraid that most sites have conditions like this so most scrapers are in fact violating terms of the site
We do allow the limited use of robots and crawlers, such as those from certain search engines, with our express written consent. If you are interested in receiving our express written permission to use robots or crawlers on our site, please contact our Licensing Department.
I'd be happy to kick-off communications. I'd think this is the type of project that they would be open to.
Spiff: Let me know if this is something I should look into, or if it's up to the XBMC team to handle.
TheUni
i think we will have to handle this ourself.
thanks for the offer though
ultrabrutal
2009-01-12, 16:19
I'm afraid we eventually have to create our own movie database ala the mymovies database which we can all contribute to :(
themoviedb.org is just that. i have already written the scraper a few months ago, but as the site had so little info i didn't bother with polishing it.
Let me quote Conan O Brien "Nicely Done" :cool:
Just to take one, imdb:
IMDB Conditions (http://www.imdb.com/help/show_article?conditions)
"Robots and Screen Scraping: You may not use data mining, robots, screen scraping, or similar data gathering and extraction tools on this site, except with our express written consent as noted below."
Do XBMC have this permission? Just curious...
ultrabrutal
2009-01-12, 18:32
Pike, when commitmonitor showed me the commits spiff had done I felt really bad. I use the scrapers also hehe. Of course XBMC cannot violate any licenses, so I totally get spiff.
spiff, I didn't know about themoviedb.org. Do you use an api or really scrape the html? I don't see an imdb link, so you use the movie name as a param?
it's an xml based api.
see r17035
Pike, when commitmonitor showed me the commits spiff had done I felt really bad. I use the scrapers also hehe. Of course XBMC cannot violate any licenses, so I totally get spiff.
spiff, I didn't know about themoviedb.org. Do you use an api or really scrape the html? I don't see an imdb link, so you use the movie name as a param?
A basic themoviedb.org scraper just hit the SVN in revision 17035. Searches and IMDB-ID translation is done via their API.
FYI, it's the new default movie scraper ..
ultrabrutal
2009-01-12, 18:49
The first thing that springs to mind when looking at the site, is... Who owns the copyright of fanart and posters? I think there is a grey area there also I'm afraid :(
offtopic: spiff, have looked into the api info. looks easy to use. any indications on speed vs the others?
LaTropa64
2009-01-12, 20:10
No more imdb scraping is going to kill one of the best things about XBMC. :(
I've added a few things to themoviedb.org when movies were missing fanart but manually entering all the actors and plot details doesn't sound like much fun. I've exported my library as nfo files though so I wonder if there's any chance we could just upload all our nfo's to a central location (like themoviedb.org) for scraping?
ultrabrutal
2009-01-12, 20:30
The nfo's are based on copyright data so they are a violation also I think.
I don't think that manually entering the data is that hard when enough people contribute. It's our best bet.
You can use mymovies.dk but it only has movies which can be bought on media
marlboroman1
2009-01-12, 20:36
I'm would realy like some one to quietly delete this thread so we can carry on in blissfull ignorance as if non of this unpleasantness occured
BIG NOTICE !
From 17033 the IMDB scraper DOES NOT WORK ANY LONGER !
they do not allow scraping their web site and XBMC is complying with this.
the new default scraper is themoviedb.org (http://themoviedb.org/) which needs help filling, so please go there and help them (and thus yourself) out :)
Does this also means that Media Companion and Movie info plus are not allowed to scrape there anymore?
My cached data is pulled from the imdb list files, it does not create an online database, is not a commercial application, it's not charging any fees, It does reference IMDB as the source of that data with links to there site.
MIP's cached data is in compliance with the imdb list file usage.
All data and software released by Internet Movie Database Ltd is
freely available to anyone within certain limitations. You are
encouraged to quote subsets of the database in USENET articles,
movie related FAQs, magazine articles etc. We do ask, however,
that you make reference to the source of the data and provide a
pointer to the database for the benefit of the reader.
MIP also does a standard http query and downloads the same full html page as any other browser, the difference is in how I display that data as I parse it into a different format again with reference to IMDB as the source of that data.
My cached data is pulled from the imdb list files, it does not create an online database, is not a commercial application, it's not charging any fees, It does reference IMDB as the source of that data with links to there site.
MIP's cached data is in compliance with the imdb list file usage.
All data and software released by Internet Movie Database Ltd is
freely available to anyone within certain limitations. You are
encouraged to quote subsets of the database in USENET articles,
movie related FAQs, magazine articles etc. We do ask, however,
that you make reference to the source of the data and provide a
pointer to the database for the benefit of the reader.
MIP also does a standard http query and downloads the same full html page as any other browser, the difference is in how I display that data as I parse it into a different format again with reference to IMDB as the source of that data.
nekrosoft13
2009-01-12, 23:25
The first thing that springs to mind when looking at the site, is... Who owns the copyright of fanart and posters? I think there is a grey area there also I'm afraid :(
offtopic: spiff, have looked into the api info. looks easy to use. any indications on speed vs the others?
can you please be quite, once you are done everything will be a violation.
jmarshall
2009-01-12, 23:38
Thanks for the heads up on the license issues. Team XBMC will contact IMDb directly and ask for their approval if they are willing to give it.
For the meantime, take the opportunity to contribute to an open database such as themoviedb. If each of us take the time (an hour or so ain't that much) to contribute 10 movies from our collections (obviously NOT just resubmitting IMDb data!) then their collection will swell extremely quickly.
After all, look at how awesome thetvdb is now, thanks to user contributions from people like you and me.
I'll post back once I hear back from IMDb.
Cheers,
Jonathan
ultrabrutal
2009-01-12, 23:55
can you please be quite, once you are done everything will be a violation.
You forgot the smiley
Thanks for the heads up on the license issues
Np I guess. Never thought that spiff would be that radical. Let's just hope imdb guys are cool
I like to know more as to why this has to change.
The main reason I use these applications is to pull data from a website such as IMDB.
They have never sent out letters, emails, or anything to other projects.
Mythtv, Meedios are not reporting any issues.
Time to time IMDB changes that function, of scrapping but the coders keep up.
Frak.
fekker Thanx for the reply and aplication.
LaTropa64
2009-01-13, 00:14
My cached data is pulled from the imdb list files, it does not create an online database, is not a commercial application, it's not charging any fees, It does reference IMDB as the source of that data with links to there site.
MIP's cached data is in compliance with the imdb list file usage.
MIP also does a standard http query and downloads the same full html page as any other browser, the difference is in how I display that data as I parse it into a different format again with reference to IMDB as the source of that data.
I'm going to agree with this. I'm essentially just using XBMC to access imdb.com instead of FireFox or Internet Explorer.
jmarshall
2009-01-13, 00:26
As I said in the other thread, Team XBMC has contacted IMDb and will leave the IMDb scraper disabled until we hear from them. Note that we were not contacted from IMDb, but are simply responding to a users' request.
It is unclear to the team as to whether we're doing anything wrong or not, so given that, the wisest action is to cease distributing the scraper until we are certain that we are doing things correctly.
In the meantime, please spend some time contributing data to themoviedb so that it's available to others. If we all spend an hour or two submitting data (not just copy + pasted from IMDb, ofcourse!) on some of our own movies, it will benefit the entire community.
Thanks for your understanding.
Cheers,
Jonathan
rodalpho
2009-01-13, 02:23
Not that any of us would knowingly violate their TOS, and I certainly didn't do it myself, but just for academic purposes, do you mind if I post how to reenable the scraper?
This is odd, oh well no more updating until it gets back.
I refuse to lose that ability for imdb :P
not saying hoorah or anything for this latest development (although it is reversable if need be), but this may lead to some speed improvements if we end up with a setup that periodically keeps a local cached copy of imdb and periodically updates it (http://www.imdb.com/interfaces).
ultrabrutal
2009-01-13, 09:55
just for academic purposes, do you mind if I post how to reenable the scraper?
Well you look at what spiff did in the revision where he disabled the scraper. Undo this to the source and rebuild
Gamester17
2009-01-13, 11:28
As I said in the other thread...For reference, here is a link to that other thread => http://xbmc.org/forum/showthread.php?t=43626
rodalpho
2009-01-13, 16:33
Yep.
marlboroman1
2009-01-13, 19:08
would it not be quicker to back up your existing imdb.xml and the overwrite the new one
smcnally75
2009-01-13, 19:19
The first thing that springs to mind when looking at the site, is... Who owns the copyright of fanart and posters? I think there is a grey area there also I'm afraid :(
offtopic: spiff, have looked into the api info. looks easy to use. any indications on speed vs the others?
For what it is worth, just the act of backing up your movies onto a hard drive violates DRM laws. The use of any photos and/or artwork without the original artists consent is a copyright violation...You pointing it out doesn't do anything but require the team to either act like they didn't see your post or remove those features to avoid getting in trouble. Without your comments, they can simply wait until they are contacted by the offended parties and asked to remove the infringing features (which may never come). If you want something like XBMC that is 100% legal then I would suggest you shell out the $20,000 for a Kaleidescape system...otherwise I would kindly ask that you stop trying to find everything that could possibly be wrong with XBMC and enjoy it.
ultrabrutal
2009-01-13, 19:51
For what it is worth, just the act of backing up your movies onto a hard drive violates DRM laws. The use of any photos and/or artwork without the original artists consent is a copyright violation...You pointing it out doesn't do anything but require the team to either act like they didn't see your post or remove those features to avoid getting in trouble. Without your comments, they can simply wait until they are contacted by the offended parties and asked to remove the infringing features (which may never come). If you want something like XBMC that is 100% legal then I would suggest you shell out the $20,000 for a Kaleidescape system...otherwise I would kindly ask that you stop trying to find everything that could possibly be wrong with XBMC and enjoy it.
So basicly what you are saying is that it's ok to "steal" as long as the person you steal from doesn't find out. If he finds out you kindly give the "stolen" goods back and all is forgotten.
The DRM laws does not apply to all countries. USA law is not global you know.
I never asked spiff to remove the scrapers. I just wanted to know if XBMC had permission. XBMC does not violate any laws more than a legal gun in your drawer does. XBMC is licensed under GPL so it wants others to respect this license right? Why should XBMC not follow others licenses? Because you don't care about licenses?
If a feature in XBMC violates a law in your country I think it's up to you to refrain from using the feature basicly.
I don't think IMDB would sue XBMC... For once, there is no money in XBMC, secondly again USA law does not apply to the world. However IMDB's business is their website and their database and not providing their data freely to anyone who wants it.
As for the fan art. Ofcourse the copyright belongs to the studio, but aren't it often based on free press kits? I can only see it as good free adverticing for the movies. The info typed into imdb's database however is another story. The info is not owned by the studios. Ofcourse a written consent about using the art work from the studios would be prefered but how would one come about it? But I think this is another story than borrowing data from imdb's database without their consent
nalthien
2009-01-13, 20:49
I would honestly like to believe this is going to be a non-issue in another week or two. Bringing it up is fine; and Team-XBMC is already talking with IMDB to secure proper permission to use their data. Should IMDB refuse (as is their right to do,) others have suggested some solid ways to resolve this problem.
I'd be frustrated to lose IMDB too--but you never know until you ask. :)
smcnally75
2009-01-13, 21:05
So basicly what you are saying is that it's ok to "steal" as long as the person you steal from doesn't find out. If he finds out you kindly give the "stolen" goods back and all is forgotten.
No, what I'm saying is that there are a lot of gray areas with stuff like this. For the most part, a lot of companies wouldn't bother giving the team a hard time about it unless someone made a stink about it. Just as the movie industry wouldn't crack down on a person in the US storing their own DVD's on a hard drive unless somebody else was crying foul and they needed to set an example. IMO, it isn't our place to bring up what could be legal and illegal. Just as you said yourself, US laws aren't global...We should leave those decisions up to the team and if any corporations get upset they will let the team know right away by ordering a cease and desist letter. I personally think calling site scraping for movie data "stealing" is a stretch. Especially when all the info on IMDB is user submitted.
ultrabrutal
2009-01-13, 21:13
No, what I'm saying is that there are a lot of gray areas with stuff like this. For the most part, a lot of companies wouldn't bother giving the team a hard time about it unless someone made a stink about it. Just as the movie industry wouldn't crack down on a person in the US storing their own DVD's on a hard drive unless somebody else was crying foul and they needed to set an example. IMO, it isn't our place to bring up what could be legal and illegal. Just as you said yourself, US laws aren't global...We should leave those decisions up to the team and if any corporations get upset they will let the team know right away by ordering a cease and desist letter. I personally think calling site scraping for movie data "stealing" is a stretch. Especially when all the info on IMDB is user submitted.
So now you want to censor me? wtf
Yes the decisions are up to the team. I just kindly asked because I was curious. I never told spiff to disable the two scrapers! Heck I use IMDB scraper too you know!
I used stealing as a parallel to what you were saying, dude. IMDB clearly states that they won't allow scrapers and robots to grab their data. If XBMC does not respect this, then they can shove their GPL up their butts. How can you ask for respect if you give out none yourself?
glances up at the subforum description:
Scraper Development Developers forum for meta data scrapers. Scraper developers only!
Not for posting feature requests or end-user support requests!
can we stop this now as this is starting to get out of hand....
natethomas
2009-01-15, 00:32
One of the fun things about the law is that each word has its own meaning. If you use the wrong word, you mean something else.
The reason I bring this up is to point out that "stealing" is not against the law in any state, to my knowledge. "Theft" of personal property (i.e. physical objects) is. However, intellectual property (code, ideas, etc.) is not personal property (once again, a physical object), which means the improper use of it is technically not theft.
What it technically is can be a whole bunch of other things, like "copyright infringement." As such, smcnally is more accurate in saying there is gray area here, since most copyright law is relatively new and/or untried in federal/state court, so none of us (being lawyers) know what any of the rules are, exactly. This is doubly true for computer-related IP, as fair-use laws and rulings are a maximum of about 35 years old.
So, just to make my point wholely clear, so long as no one is making a profit or causing others to massively lose profit, odds are no one is going to sue anyone else, because the cost of figuring out exactly what the law is in court is COMPLETELY not worth it.
Of course, with that said, asking permission is always a good idea.
theophile
2009-01-16, 02:07
X2
IMDB cannot claim intellectual property rights over mere factual data. Titles and cast lists of movies are mere factual data and most of the plot synopses are user-contributed, so IMDB has no claim of right over them anyway, unless there is a clause saying that any user-contributed data becomes the intellectual property of IMDB.
Even so, the fact that XBMC is rearranging and reformatting the data could qualify it for fair use exceptions even if the data were intellectual property.
In short, a website's TOS is not a legally binding document. IMDB's policy on screen-scraping amounts to nothing more than a request which users may decline at their option.
theophile
2009-01-16, 02:11
If XBMC does not respect this, then they can shove their GPL up their butts. How can you ask for respect if you give out none yourself?
The GPL is a software license. It is a legally binding document. A TOS is not, unless a user is explicitly required to accept it in order to use the website. If it does, it becomes a license.
License = contract
Terms of use /= contract
ultrabrutal
2009-01-16, 09:34
Maybe you are right. I have no idea. I'm not a lawyer or judge and whether or not they own the copyright of the data in their database is not up to me to judge. Let's hope you are correct, but I think that the team needs alittle more solid ground inother to reenable the disabled scrapers. Can you provide this? That would be great thanks
theophile
2009-01-16, 17:51
Many web sites make you click on “agree” to the terms and conditions before going on, but Ticketmaster does not. Further, the terms and conditions are set forth so that the customer needs to scroll down the home page to find and read them. Many customers instead are likely to proceed to the event page of interest rather than reading the “small print.” It cannot be said that merely putting the terms and conditions in this fashion necessarily creates a contract with any one using the web site.
Ticketmaster Corp. v. Tickets.Com, Inc., 2000 WL 525390, 3 (C.D.Cal., 2000).
[A]n offeree, regardless of apparent manifestation of his consent, is not bound by inconspicuous contractual provisions of which he was unaware, contained in a document whose contractual nature is not obvious....
Windsor Mills, Inc. v. Collins & Aikman Corp., 25 Cal.App.3d 987, 993, 101 Cal.Rptr. 347 (Cal.Ct.App.1972).
Netscape argues that the mere act of downloading indicates assent. However, downloading is hardly an unambiguous indication of assent. The primary purpose of downloading is to obtain a product, not to assent to an agreement. In contrast, clicking on an icon stating “I assent” has no meaning or purpose other than to indicate such assent. Netscape's failure to require users of SmartDownload to indicate assent to its license as a precondition to downloading and using its software is fatal to its argument that a contract has been formed.
Specht v. Netscape Communications Corp., 150 F. Supp. 2d 585, 595 (S.D.N.Y., 2001).
To summarize, users of IMDB are not bound by the terms of usage of the website because they are not required to accept them. I was unable to make the moderators of Mythbuntu forums see this fact when it came to a program to obtain EPG data from Microsoft Media Center servers. The Microsoft TOS for their Media Center data explicitly stated that it could not be accessed with anything other than an authorized Microsoft product, but that does not apply to anyone who does not explicitly accept that TOS/license.
If I had purchased and installed Vista MCE (for example), I would have been required to explicitly accept the terms of the license (click "I Agree" in order to install it). IN that case, the terms would be binding. But as a Linux user I've never accepted Microsoft's TOS and am therefore not bound by them. That means I can access data from Microsoft's EPG servers any way I like and there are no legal ramifications. If Microsoft wants to make their servers inaccessible to anything other than a Microsoft product, it's up to them to do so.
In like manner, if IMDB wants to make its users explicitly agree not to scrape the website or to make the website "unscrapable," it's up to them to do so. In the mean time, as I said, their TOS amounts to nothing more than a request. It's not enough to form a contract and it is not legally binding, even if you HAVE read it and know what it says.
I don't understand why you guys are trying to turn this into a legal entitlement issue. XBMC is an opensource project that depends on the cooperation and willingness of users/devs/affiliates/etc.
XBMC could probably scrape from the imdb site no problem, even if they explicitly forbade it. But that's not the issue. The issue is that they've asked that it not happen without consent. Free access to a huge movie database and all they want is for you to ask... sounds fair to me. If they decline, the community will organize something new.
If too many projects scraped their site they would be forced to block it somehow. They would have every right. Again, it's not an issue of entitlement, it's about being fair and cooperating.
Let's try to be civil and not demanding. I think the XBMC team is doing it just right.
TheUni
natethomas
2009-01-16, 20:53
I don't think anyone was saying we are legally entitled to IMDB's info and should, therefore, take it as we please. Instead, ultrabrutal made the point that to do so would be "stealing," and people were irritated by that minor inaccuracy, since no one likes to be called a thief. Thus, we engaged in a brief legal discussion on the definition of breach of contract and infringement.
I personally agree that getting the permission of IMDB would be the way to go, since cooperation in this area could lead to improved scraping times, reduced bandwidth usage, and a generally more pleasant experience, or, in the alternative, the improvement of an alternative "open" website for scraping purposes would do the same thing.
With that said, having a frank and open discussion of the state of the law is never a bad thing. If nothing else, many times it's a good way to find out that there are efficient, legal ways to accomplish goals, and, in our current predicament, it's a good way to find out that, even if we do decide to go down a different path in the future, our previous actions were still well within the law and entirely reasonable.
TomJensen
2009-01-16, 21:31
I don't understand the rationale for the stopped distribution of the IMDB scraper. As said, IMDB didn't initiate contact with a cease&desist, it's actually XBMC that asked. What organization in their right mind would say that it's OK to use their content, w/o the benefit of bannering their ads? No one. I'll bet if XBMC starts asking any of the other content sites, most of them would give a thumbs down. Suddenly elevating this to a cease&desist seems overly paranoid and self-defeating to me. Would Apple be happy and give permission to distribute the Apple trailer plugin? May be we should ask them? How about YouTube? How about any of the content orgs that XBMC has a scraper or plugin for?
Let's be clear, as an open-source app that bypasses most "official" (read: DRM'ed or ad-driven) solutions, no company is going to approve of XBMC. If XBMC gets a cease&desist lawyerese form, fine, it should stop distrib the scraper. But otherwise, keep on trucking. At the least, leave the scraper in the distribution, but don't enable it as a default.
iluvXBMC
2009-01-16, 21:33
Hi Spiff,
I was just wondering when you were going to stop scraping allmusic.com since it's against their TOS to scrape. Their TOS found here (http://allmusic.com/cg/amg.dll?p=amg&sql=32:amg/info_pages/a_terms_of_service.html) states: "You may not scrape or otherwise copy our Material without permission." Likewise, when does XBMC plan to stop allowing access to Hulu content? Their TOS found here (http://www.hulu.com/terms) states that you may not scrape their site or playback any content outside of the Hulu video player. You sure you want to start down this slippery slope?
jmarshall
2009-01-16, 22:20
Thread closed - it's gone too far off topic.
Like I said before this bongles my head.
It makes no sense why and I'm not even going to get into it.
Everyone use IMDB for that reason.
Example:
MythTV
Meedios
Mediaportal
GBPVR
Sagetv
I can make this list a lot bigger, none them are running around stopping it.
Whatever I'm grumpy today.
I don't think we need to debate the devs decision to remove IMDB (unless they want us to ;)), but I think that it would be wise to carefully consider the alternatives and make sure there won't be any problems down the line. I think the two prerequisites to consider before you contribute and volunteer your time to any site should be:
1) is the content "free"
and
2) is there a database dump available
This way it is to fork if the site becomes defunct or "evil". I still remember the uproar when CDDB took people's work and became commercial and hopefully we can avoid mistakes like that in the future.
thetvdb.com uses the Creative Commons "Attribution" license, which is perfect. themoviedb.org also uses the Creative Commons licens, but with the "Attribution" and "Non-commercial" clauses. IMHO, the "Non-commercial" clause might be problematic, but is still better than IMDB.
TomJensen
2009-01-17, 02:52
@marirs
I think you are going off on a tangent. The issue isn't about supporting "free" alternatives; it is about whether XBMC should use the content of any site without expressed permission of the site owner. As related, I highly doubt any site owner would be willing to do that, which means that, if XBMC were to make this a policy--no content w/o explicit permission--then we should all turn off the lights and go home, because there won't be any online content that XBMC will be allowed to access.
As another said, it is a slippery slope when you say that because of the TOS, we can't have the IMDB scraper. Most every site I know of has policies in its TOS against scraping or redirection. Video plugins are even more questionable than scrapers, as substantial bandwidth is being used, but without compensation from ad revenue.
What needs to be done is to disassociate XBMC from any legal liability, either by offloading the liability onto the user ("these are provided for educational purposes only" :), or by separating plugins & scrapers from XBMC and making the user manually installing it--eg, provide a link, with appropriate indemnification legalese. But whatever is decided, it should be a policy, consistent to all utilized online resources and not just arbitrary ones.
It seems that the discussion whether XBMC is doing "illegal" things (accessing other sites' content w/o their consent) has been relegated to the dumper. Hey, the deaf-dumb-blind approach certainly works for me. But this is an issue that will rear its head as apps like XBMC get more popular.
Here's a NY Times article on Boxee (XBMC derivative), which is asking the same question the users here were asking (accessing sites' content). It's well worth the read:
http://www.nytimes.com/2009/01/17/technology/internet/17video.html?em
This is a discussion that ought to happen now, i.e. whether the TOS'es (which users never expressly agreed to when they get onto sites) are legally binding, or do they have the same status as shrinkwrap agreements--read: not very much. My opinion is the latter. In lieu of that, the "ignorance is bliss" defense was certainly fine, and dropping the IDMB scraper (while keeping everything else) is nothing if not capricious.
Frankly, I see the killing of IDBM lookup capability as nothing more than a power play by the powers-that-be to get more attention for the freebie alternative. That's fine. XBMC is their toy and they can put it out in whatever shape they please. But do realize that the end result is XBMC's capability being degraded, by being shunted to an inferior scraper.