tag:blogger.com,1999:blog-8592569895405724581.post8249983148533477773..comments2011-11-18T23:58:22.854-08:00Comments on Algorithm Analysis: BellKor Algorithm: Pearson Correlation CoefficientLarry Freemanhttp://www.blogger.com/profile/06906614246430481533noreply@blogger.comBlogger29125tag:blogger.com,1999:blog-8592569895405724581.post-3119337634207168112010-08-23T07:17:32.452-07:002010-08-23T07:17:32.452-07:00Hi Amel,
Remember a result is between 1 and -1. ...Hi Amel,<br /><br />Remember a result is between 1 and -1. 0 Means that the two results are not as similar as they could be (1 is closest) and they are not as dissimilar as they could be (-1 is the farthest).<br /><br />-LarryLarry Freemanhttps://www.blogger.com/profile/06906614246430481533noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-33362092865240640252010-08-23T01:59:46.621-07:002010-08-23T01:59:46.621-07:00Hi Larry,
Thank you for this blog !
T have a qu...Hi Larry,<br /><br /><br />Thank you for this blog !<br /><br />T have a question about the pearson correlation formula:<br /><br />When I compute correlation between 2 movies evaluated by 2 users <br /><br />Exemple: <br />user 1 gives following ratings : 1, 1, 1 to movie : M1, M2, M3.<br />user 2 gives following ratings : 1,1,1 to movies: M1, M2, M3.<br /> <br />the result of the pearson correlation formula is 0. <br /><br />So How I could interprate this result?<br /><br />Thank for your helps.<br /><br />Amel.Amelhttps://www.blogger.com/profile/02983463761756947824noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-3925178028655160932009-06-11T14:27:57.814-07:002009-06-11T14:27:57.814-07:00Hi Newman!,
Yes, a typo. I meant, I improved by ...Hi Newman!,<br /><br />Yes, a typo. I meant, I improved by .0005.<br /><br />I've been thinking about RMSE with a multiple of 1000.<br /><br />I haven't submitted my implementation of the BellKor knn yet. I am hoping to do that tonight.<br /><br />I'll be very glad to review your blog entry.<br /><br />Cheers,<br /><br />-LarryLarry Freemanhttps://www.blogger.com/profile/06906614246430481533noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-86485135598459705962009-06-11T12:10:06.455-07:002009-06-11T12:10:06.455-07:00Hi Larry,
There's a typo here: "My stand...Hi Larry,<br /><br />There's a typo here: "My standard version of KNN improved my quiz RMSE by 5." ? <br /><br />I just finish a blog on my optimizations of KNN calculations. I'd appreciate it if you take a quick look and tell me if some part is not clear ?dmnewbiehttps://www.blogger.com/profile/14537616692777428127noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-9389765770205407872009-06-06T21:50:51.479-07:002009-06-06T21:50:51.479-07:00Hi Newman!,
My standard version of KNN improved m...Hi Newman!,<br /><br />My standard version of KNN improved my quiz RMSE by 5.<br /><br />KNN combined with SVD++, etc. gave me a quiz score of .8848.Larry Freemanhttps://www.blogger.com/profile/06906614246430481533noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-53665325543184308382009-05-11T20:17:00.000-07:002009-05-11T20:17:00.000-07:00Hi Newman!,
I've haven't completed the KNN algori...Hi Newman!,<br /><br />I've haven't completed the KNN algorithm yet so I don't have an answer for you.<br /><br />Recently, I've been working with SVD, SVD++, and the Brism.<br /><br />I am planning to finish my work with KNN after I get SVD++ working.<br /><br />-LarryLarry Freemanhttps://www.blogger.com/profile/06906614246430481533noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-85429575414823662862009-05-11T18:36:00.000-07:002009-05-11T18:36:00.000-07:00Larry, what's the probe and qualifying RMSE of you...Larry, what's the probe and qualifying RMSE of your KNN algorithm ?dmnewbiehttps://www.blogger.com/profile/14537616692777428127noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-25882762822452503232009-05-03T08:43:00.000-07:002009-05-03T08:43:00.000-07:00Oh! I see - the +10 is the alpha value for the shr...Oh! I see - the +10 is the alpha value for the shrinking of the pearson. Sorry, missed that bit. I couldn't work out why comparing a movie against itself wasn't coming out with 1.0 with that in there - but I saw after that without this, you get lots of 1.0 values for movies with low support.<br /><br />Sorry for spamming your blog with my noobishness, Larry.f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-https://me.yahoo.com/a/f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-#86d2dnoreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-15094558436271436462009-05-02T10:50:00.000-07:002009-05-02T10:50:00.000-07:00Hi Larry,
Ok, I finally got this up and running f...Hi Larry,<br /><br />Ok, I finally got this up and running for me, so I thought I'd post a link to the complete job:<br /><br />http://pastebin.com/f315f7bcb<br /><br />This includes all the server access garb, some time outputs to let you see the progress, and FileWriter outputs. I ran this from Run in Eclipse as, not knowing anything about Java, I couldn't get it to compile from the command line ;)<br /><br />As I said before, I had to change a couple of the k & l iterators. In addition, in order to get sane results I had to change sumsq1, sumsq2 and sumpr to += and deleted the +10 from "float pearson = (top/bottom)*(num/(num+10))".<br /><br />Many thanks for all your work - I'd still be nowhere without it. <br /><br />Andyf5U5uht2i_c3FPczR6J9_f2GQPzr3n0-https://me.yahoo.com/a/f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-#86d2dnoreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-22088685131919038762009-04-29T11:26:00.000-07:002009-04-29T11:26:00.000-07:00err, anyone else trying to use the code I pasted s...err, anyone else trying to use the code I pasted should probably remove the 3 "System.out.println(+new Date());" lines if you're running from the command line. I've also limited the final sql rating query to 0,1000000 to make the loadup quicker for debugging, so remove that. This has no apparent effect on the exception.f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-https://me.yahoo.com/a/f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-#86d2dnoreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-84719272901933029212009-04-29T11:18:00.000-07:002009-04-29T11:18:00.000-07:00Hi Larry,
Thank you for your further explanation,...Hi Larry,<br /><br />Thank you for your further explanation, I think I see what I'm supposed to do; but I'm still not sure why your code is stopping at 8,17770. I've posted a copy here with all the sql access stuff included.<br />http://paste.pocoo.org/show/115007/<br /><br />The only changes I've made are to correct some typos, and these lines:<br /><br />"for (int k=userIndex[relUserId]; k < userIndex[relUserId+1]; j++)" - where I changed j++ to k++. And:<br /><br />for (int l=1; k <= 5; l++) where I changed k to l.<br /><br />Are these corrections correct?<br />The exception comes at 8,17770 when trying to update the "values[movieByUser[k]-1][ratingByUser[k]-1][ratingByMovie[j]-1]++;"f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-https://me.yahoo.com/a/f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-#86d2dnoreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-50600216571289799142009-04-28T15:39:00.000-07:002009-04-28T15:39:00.000-07:00Sorry for the problems with the code. I have not ...Sorry for the problems with the code. I have not figured out an easy way to get code into Blogger. It insists on reformatting it.<br /><br />Here's the essence of the code:<br /><br />(1) int[] usersOrderdByMovie<br /><br />Assume this is an array of users ordered by movie.<br /><br />(2) int[] movieIndex<br /><br />Assume that this is an index that maps movies to usersPerMovie.<br /><br />(3) int[] numUsersPerMovie<br /><br />This is a number of users who rated a movie.<br /><br />In this way, we can iterate through all users for a movie 10 by using:<br /><br />int base=movieIndex[10];<br />int n = numUsersPerMovie[10];<br />for (int i=base; i < base+n; i++) {<br /> int userId = usersOrderedByMovie[i];<br />}<br /><br />We can likewise do the same for users so that we have:<br /><br />(4) int[] moviesOrderedByUser<br /><br />(5) int[] userIndex<br /><br />(6) int[] numMoviesPerUser <br /><br />Now here's the algorithm for calculating the Pearson for all movies:<br /><br /> <br />double[] similarity = new double[NMOVIES];<br /><br />short[][][] values = new short[NMOVIES][5][5];<br /><br />for (int movieid=1; movieid < 17771; movieid++) {<br /><br />for each user (u) who rated this movie {<br /><br /> for each other movie (m != movieid) rated by this user {<br /><br /> let r = rank for u,m<br /> values[m][u][r]++; <br /><br /> }<br /><br />}<br /><br /><br />}<br /><br />That's the essence of the algorithm. The tally is just a count to make it easier to order the usersOrderedByMovie and the moviesOrderedByUser.<br /><br />-LarryLarry Freemanhttps://www.blogger.com/profile/06906614246430481533noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-67360833894314420482009-04-28T14:19:00.000-07:002009-04-28T14:19:00.000-07:00I managed to get the code up and compiled, after c...I managed to get the code up and compiled, after correcting several apparent mistakes with the iterators. But I'm getting some kind of exception at the 8,17770 mark, and the pearson results don't look at all right. Is there any chance of this code being updated to something usable? The whole tally part mystifies me and I just can't work it out :/f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-https://me.yahoo.com/a/f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-#86d2dnoreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-88339703690142172082009-04-17T23:54:00.000-07:002009-04-17T23:54:00.000-07:00Just realized that Larry answered my question in h...Just realized that Larry answered my question in his previous blog. I.e. you do rating minus all the effects, then predit using your knn algorithm and then add the effects back in to produce the qualifying result. This looks reasonable.MikeMhttps://www.blogger.com/profile/05411310108810172734noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-87311861866683072432009-04-15T13:57:00.000-07:002009-04-15T13:57:00.000-07:00MikeM, I might be wrong, I haven't read Mike's oth...MikeM, I might be wrong, I haven't read Mike's other page. But the idea I got from reading the papers was that you'd run the pearson on the data set after you've applied the global effects - they are not predictive models in and of themselves. The GE iron out anomolies and bumps in the data, the pearson ties them together.f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-https://me.yahoo.com/a/f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-#86d2dnoreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-46149967114716921092009-04-15T12:32:00.000-07:002009-04-15T12:32:00.000-07:00It is not clear to me how global effects is used i...It is not clear to me how global effects is used in the pearson calculation, i.e. no use is made of 'residual'. I'm about to do #4 of your previous blog and came to a grinding halt when I realized that this is not used in the pearson calculation at all described in this blog. Am I missing something?MikeMhttps://www.blogger.com/profile/05411310108810172734noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-57404754596955417652009-04-11T03:45:00.000-07:002009-04-11T03:45:00.000-07:00That would be great, Larry. Thank you again.That would be great, Larry. Thank you again.f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-https://me.yahoo.com/a/f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-#86d2dnoreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-72767185831172675592009-04-10T14:22:00.000-07:002009-04-10T14:22:00.000-07:00Hi Andy,Thanks for your feedback.I'll try to put a...Hi Andy,<BR/><BR/>Thanks for your feedback.<BR/><BR/>I'll try to put all the java code in a single place in the next week or so. I'll post the link here when I do.<BR/><BR/>-LarryLarry Freemanhttps://www.blogger.com/profile/06906614246430481533noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-61669847054382004392009-04-10T14:09:00.000-07:002009-04-10T14:09:00.000-07:00Hi Larry,Many thanks for this code - I also used y...Hi Larry,<BR/><BR/>Many thanks for this code - I also used your sql setup and it worked great. I don't know much about java and I'm having trouble compiling your code. Is there any chance you could post it as a single file?<BR/><BR/>- Andyf5U5uht2i_c3FPczR6J9_f2GQPzr3n0-https://me.yahoo.com/a/f5U5uht2i_c3FPczR6J9_f2GQPzr3n0-#86d2dnoreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-1273646188177211312009-02-04T02:42:00.000-08:002009-02-04T02:42:00.000-08:00hi, thank you very much for your writeups!FYI... i...hi, thank you very much for your writeups!<BR/><BR/>FYI... if you run the knn on residuals from the global effects, adventures of robin hood suddenly has very reasonable continuous-like neighbors. that is--bonus mt'l, pirates of caribbean, indep day, lor 123, american beauty, and so on. I tried to email you, but can't find your email anywhere.Greghttps://www.blogger.com/profile/16611329016300933519noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-16163357815211026302008-10-22T20:59:00.000-07:002008-10-22T20:59:00.000-07:00Yes, that's a well known problem with MySQL's vers...Yes, that's a well known problem with MySQL's version of JDBC. I'm glad that you figured it out.<BR/><BR/>Here's the <A HREF="http://bugs.mysql.com/bug.php?id=18148" REL="nofollow">bug report</A>.<BR/><BR/>-LarryLarry Freemanhttps://www.blogger.com/profile/06906614246430481533noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-8749300467262304772008-10-22T20:06:00.000-07:002008-10-22T20:06:00.000-07:00Thanks for the quick response. I actually just fi...Thanks for the quick response. I actually just figured out the problem was indeed that JDBC was trying to load the entire ratings query result into memory. Statement.setFetchSize(1) didn't solve my problem, but I learned that strangely Statement.setFetchSize(Integer.MinValue) did! Unless this is some strange behavior by my version of the JDBC drivers, I'm not sure why this works.<BR/><BR/>-TaylorTaylorhttps://www.blogger.com/profile/04768115400776434273noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-48810301555189007942008-10-22T08:16:00.000-07:002008-10-22T08:16:00.000-07:00How much memory does your machine have? Also, are...How much memory does your machine have? Also, are you running any other applications at the same time? For example, I found that I needed to close down Firefox to get it working.<BR/><BR/>I was able to run my algorithm on a machine with 1GB memory.<BR/><BR/>-LarryLarry Freemanhttps://www.blogger.com/profile/06906614246430481533noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-90676358637633807942008-10-21T12:16:00.000-07:002008-10-21T12:16:00.000-07:00I've been following your blog and loved the level ...I've been following your blog and loved the level of detail you covered on this topic. I tried to reproduce your results, but am getting OutOfMemoryErrors before my query can get all the data from the ratings table. I tried doubling your recommended heap size and that didn't do it. I even tried using stmt.setFetchSize(num_rows) which suprisingly didn't fix my problem either.<BR/><BR/>Any ideas?<BR/><BR/>Regards,<BR/>TaylorTaylorhttps://www.blogger.com/profile/04768115400776434273noreply@blogger.comtag:blogger.com,1999:blog-8592569895405724581.post-52076892090345806882008-10-21T12:13:00.000-07:002008-10-21T12:13:00.000-07:00This comment has been removed by the author.Taylorhttps://www.blogger.com/profile/04768115400776434273noreply@blogger.com