Will I "Really Like" this Movie?

Navigating Movie Website Ratings to Select More Enjoyable Movies

Archive for the month “April, 2016”

Rating Movies: If You Put Garbage In, You’ll get Garbage Out

In my prior life, I would on occasion find myself leading a training session on the predictive model that we were using in our business. Since the purpose of the model was to help our Account Executives make more effective business decisions, one of the points of emphasis was to point out instances when the model would present them with misleading information that could result in ineffective business decisions. One of the most basic of these predictive model traps is that it relies on data input that accurately reflects the conditions being tested in the model. If you put garbage into the model, you will get garbage out of the model.

Netflix, MovieLens, and Criticker are predictive models. They predict movies that you might like based on your rating of the movies you have seen. Just like the predictive model discussed above, if the ratings that you input into these movie models are inconsistent from movie to movie, you increase the chances that the movie website will recommend to you movies that you won’t like. Having a consistent standard for rating movies is a must.

The best approach to rating movies is a simple approach. I start with the Netflix guidelines to rating a movie:

  • 5 Stars = I loved this movie.
  • 4 Stars = I really liked this movie.
  • 3 Stars = I liked this movie.
  • 2 Stars = I didn’t like this movie.
  • 1 Star = I hated this movie.

When I’ve used this standard to guide others in rating movies, the feedback has been that it is an easily understood standard. The primary complaint has been that sometimes the rater can’t decide between the higher and lower rating. The movie fits somewhere in between. For example, “I can’t decide whether I “really like” this movie or just “like” it. This happens enough that I’ve concluded that a 10 point scale is best:

  • 10 = I loved this movie.
  • 9 = I can’t decide between “really liked” and “loved”.
  • 8 = I really liked this movie.
  • 7 = I can’t decide between “liked” and “really liked”.
  • 6 = I liked this movie.
  • 5 = I can’t decide between “didn’t like” and “liked”.
  • 4 = I didn’t like this movie.
  • 3 = I can’t decide between “hated” and “didn’t like”.
  • 2= I hated this movie.
  • 1 = My feeling for this movie is beyond hate.

The nice thing about a 10 point scale is that it is easy to convert to other standards. Using the scales that exist for each of the websites, an example of the conversion would look like this:

  • IMDB = 7  (IMDB uses a 10 point scale already)
  • Netflix = 7 /2 = 3.5 = 4 rounded up.  (Netflix uses 5 star scale with no 1/2 stars)
  • Criticker = 7 x 10 = 70 (Criticker uses 100 point scale).
  • MovieLens = 7 /2 = 3.5 (MovieLens has a 5 star scale but allows input of 1/2 star)

Criticker, being on a 100 point scale, gives you the capability to fine tune your ratings even more. I think it is difficult to subjectively differentiate, for example, between an 82 and an 83. In a future post we can explore this issue further.

So from one simple evaluation of a movie you can generate a consistent rating across all of the websites that you might use. This consistency allows for a more “apples to apples” comparison.

So throw out the garbage. Good data in will produce good data out, and a more reliable list of movies that you will “really like”.



Rotten Tomatoes: When Fresh is Not So Fresh

What do Mary Poppins, The African Queen, Sleepless in Seattle, In the Heat of the Night, An Officer and a Gentleman, The Natural, and Mr. Smith Goes to Washington all have in common? They are a few of the many Academy Award nominated movie classics that do not earn the Rotten Tomatoes Certified Fresh designation. The reason they aren’t Certified Fresh isn’t because movie critics didn’t like these movies. All of these movies have been graded Fresh by at least 75% of the critics who have reviewed the movies. They fail the Certified Fresh test because not enough of Rotten Tomatoes’ critics have reviewed these movies. They fall into a hole in the Rotten Tomatoes ratings system.

Rotten Tomatoes established their website in 1998. Movie critics for the most part review today’s movies. So the movies with the most reviews used in the Rotten Tomatoes ratings are for those released after 1997. Some of the original Rotten Tomatoes critics, Roger Ebert for example, had been writing reviews for a number of years and had also reviewed some of the classic movies. These few critics gave some of the older movies some reviews in Rotten Tomatoes. But over the years, there haven’t been enough critic reviews of the pre-1998 movie releases to meet the 40 review minimum for limited release movies and 80 review minimum for wide release movies. The movies that would otherwise be graded Certified Fresh except for their failure to meet the minimum critic reviews are graded as Fresh.

There are legitimate reasons for having these minimums. Having 75% of the critics grade a movie as Fresh could be as much about randomness as quality. There is no denying, though, that the movies mentioned above and many more like them would be Certified Fresh if they had been released after 1997.

Does it matter that these movies fall through the Rotten Tomatoes cracks? Yes, if Certified Fresh is your threshold for watching a particular movie. There is a whole universe of movies of higher quality that you are missing, many of much higher quality than some of today’s Certified Fresh movies. Because I use Certified Fresh as criteria for a recommended movie, it biases my movie selections to more recent films.

What can we do about it? How do we determine if the pre-1998 movie is of Certified Fresh quality and not some movie that is randomly Fresh? One idea I’ve come up with is to treat pre-1998 movies graded as Fresh, that were also nominated for an Academy Award, as a Recommended Movie. There are 418 movies released since 1998 that have been nominated for an Academy Award that are in my database. 80% of those nominated movies have been graded by Rotten Tomatoes as Certified Fresh. In contrast,  the remaining 718 non-nominated post-1998 movies in my database were Certified Fresh only 37.5% of the time. There is clearly a strong correlation between an Academy Award nomination and a Certified Fresh rating. When I change my definition of a Rotten Tomatoes Recommended Movie to include pre-1998, Academy Award nominated movies that were graded 75%+ Fresh by Rotten Tomatoes, the probability that I will “really like” the Rotten Tomatoes Recommended Movie goes from 82% to 86%.

This seems to me to be a viable alternative to the not so Fresh Rotten Tomatoes conundrum. What do you think?


When It Comes to Movie Rating Websites, There is Strength in Numbers.

If you can only use one website to help you select movies that you will “really like”, which should you choose? That’s a tougher question than you might think. Because I have used all five of the websites recommended here to select movies to watch, my data has been heavily influenced by their synergy. I have no data to suggest how effective using only one site would be. Here’s what I do have:

Probability I Will “Really Like”
Recommendation Standard When Recommended in Combination with Other Sites When Recommended by This Site Only
MovieLens > 3.73 70.2% 2.8%
Netflix > 3.8 69.9% 8.4%
Criticker > 76 66.4% 10.1%
IMDB > 7.4 64.1% 0.3%
Rotten Tomatoes Certified Fresh 62.7% 4.3%

When MovieLens recommends a movie, in synergy with other websites, it produces the highest probability. When Criticker recommends a movie but the other four sites don’t recommend the movie, then Criticker has the highest probability. Netflix is second in both groups. Which one is the best is unclear. What is clear is that the three sites that recommend movies based on your personal taste in movies, MovieLens, Netflix, & Criticker, outperform the two sites that are based on third party feedback, Rotten Tomatoes and IMDB. When Netflix, MovieLens, & Criticker recommend the same movie there is an 89.9% chance I’ll “really like” it. When both IMDB & Rotten Tomatoes recommend the same movie the probability is 75.8% I’ll “really like” it.

What also is clear is that if four websites are recommending that you don’t watch a movie and one is recommending that you do, the probability is that you won’t “really like” the movie no matter how good that one website is overall. The progression of probabilities in the example below gives some perspective of how combining websites works:

Websites Recommending a Movie Probability I Will “Really Like”
None 3.9%
Netflix Only 8.4%
Netflix & MovieLens Only 31.9%
Netflix, MovieLens, & Criticker Only 50.9%
Netflix, MovieLens, Criticker & IMDB Only 71.1%
All Five 96.6%

Stated Simply, your odds increase with each website that recommends a particular movie. If, for example, you were to only use Netflix for your movie recommendations, the probability of “really liking” a movie might be 69.9% but, in reality, it could be any one of the probabilities in the table above with the exception of the 3.9% for no recommendations. You wouldn’t know if other websites had recommended the movie.

So, if I had to choose one website, I’d choose Netflix-DVD if I were one of their 5,000,000 DVD subscribers. If I’m not already a subscriber I’d go with MovieLens. It would be a reluctant recommendation, though, because the strength in numbers provided by using multiple websites is just so compelling.


You’ll notice in the Top Ten Movies Available to Watch This Week that there are a number of movies on the list that are available on Starz. I’m taking advantage of the Comcast Watchathon Week which provides for free Starz, HBO, & Cinemax. Some of my highly rated movies which would ordinarily be unavailable are available for the short duration of this promotion. Bonus movies. Wahoo!!



Hollywood Has an even Deeper Diversity Problem than it Thinks

Children should be seen and not heard. This is a proverb whose origins date back to medieval times. It is a proverb that is rarely used today because, well, it’s so medieval. When it comes to roles for actresses in major motion pictures, however, we aren’t far removed from those medieval times. Actresses are seen in the movies but are not heard as much as their male counterparts. According to a study released within the last few weeks by the website Polygraph Cool, actresses have less dialogue than male actors by a significant amount in 2,000 of the top grossing box office films of all time. The study measures words of dialogue for each character in the screenplays of these movies. Some of the key findings in the study are:

  • Female characters have the most dialogue in only 22% of the films studied.
  • Female characters have two of the top three roles in only 18% of the movies.
  • In all age groups, actresses have less dialogue than male actors in the same  age group.
  • This dialogue discrepancy gets more pronounced as actresses age. Actresses 22-31 have 3 words of dialogue for every 4 words for Actors in the same age group. In comparison Actresses 42-65 have 1 word of dialogue for every 5 words of male dialogue.
  • Even in Romantic Comedies the dialogue is 58% male.

Are there movies out there with greater gender parity? If so, how do you find them? The answer is yes. They do exist but not in great numbers. At the bottom of the Polygraph study linked above, the authors provide a tool that you can use to access the movies used in the study. As I’ve mentioned in a prior article, there is a male bias to both IMDB and Rotten Tomatoes. You might check out ChickFlix.net, which provides movie reviews from a female perspective as a substitute for Rotten Tomatoes.

There is also the Bechdel Test, which is cited in the Polygraph study. This tests movies based on a simple criteria. There must be two main female characters who have a conversation with each other about something other than a man. Based on studies, only about 50% of movies pass the test.

You can also use the personalized movie recommenders that I’ve recommended on my posts. By rating movies on Netflix-DVD, MovieLens, or Criticker, you will generate movie recommendations based on your taste in movies.

The lack of diversity in today’s movies reflect the box office. The first step is being able to identify which movies reflect the diversity that we’d like to see in film. I would like to think that we can push film evolution out of medieval times.

Who is Your Favorite Actor or Actress? Criticker Knows.

Who is your favorite actor or actress?  If you can’t wait for the next Leonardo DiCaprio or Jennifer Lawrence movie, does that make them your favorite actors?  If you have rated on Criticker every movie you’ve ever seen, or in my case every movie seen in the last 15 years, the answer to these questions is just a click away.

Criticker has a number of neat tools on its website. One of my favorites is its Filmmaker List, which can be found by clicking the Explore button that appears along the top banner. You can rank  Actors, as well as Directors or Screenwriters, using a variety of criteria. I like to rank actors based on the average rating I’ve given the movies that they’ve  appeared in. Once you’ve ranked them, you can click on their name and see which of their movies you’ve seen and which ones you haven’t. You can also set a minimum number of movies for your rankings so that you can get the most representative sample that your number of ratings will allow.

For example, I have 1,999 movies rated in Criticker. If I set my minimum at 20 movies for Actors and rank them by average score, my top five favorite Actors are:

Actor Avg. Score # of Movies Seen Best Movie Not Seen
Tom Hanks 85.88 26 Cloud Atlas
Harrison Ford 83.50 24 The Conversation
Morgan Freeman 82.50 22 The Lego Movie
Phillip Seymour Hoffman 81.18 22 Boogie Nights
Samuel L. Jackson 81.00 25 Kingsman: The Secret Service

Based on a 15 movie minimum, my favorite Actresses are:

Actress Avg. Score # of Movies Seen Best Movie Not Seen
Kate Winslet 79.13 15 Hamlet (1996)
Scarlett Johansson 75.52 21 Hail, Caesar!
Judi Dench 74.22 17 Hamlet (1996)
Laura Linney 74.63 16 Mr. Holmes
Natalie Portman 74.35 17 Paris, je t’aime
Meryl Streep 74.28 18 Fantastic Mr. Fox

I included 6 on my Actress list because you just can’t leave Meryl Streep off of any list of the Best Actresses. Also, the Best Movie not seen is based on the highest predicted Criticker Score for movies I haven’t seen, or haven’t seen in the last 15 years.

There are a couple of surprises on my lists. Samuel L. Jackson is a surprise. I can’t say that I watch a particular movie because Samuel L. Jackson is in it. It does, however reflect how many quality movies he’s been in. Scarlett Johansson is another surprise. It’s amazing that I have seen 21 movies of hers and she is only 31 years old.

There are favorite actors of mine who didn’t make the list, such as Paul Newman and Jodie Foster. In Paul Newman’s case, he didn’t meet the 20 minimum and his average movie score wasn’t high enough (79.32 in 19 movies). Jodie Foster would have been the highest on my list with an average score of 79.64 but I’ve only seen 11 of her movies, under the 15 movie minimum I set.

When you first go to the Filmmaker List the default for the minimum movies rated is 3. Under that criteria my favorite actor of all time is Billy Dee Williams (95.67 and 3 movies). “I love Brian Piccolo”, and Lando Calrissian (Star Wars V & VI) as well.




According to IMDB, Do We Get More Conservative as We Age or Just Smarter

We are in the midst of a hotly contested election season in which the pundits attempt to place groups of people in defined boxes. Two of the boxes are the liberal youth vote and the conservative Senior Citizen vote. Some pundits argue that as Senior Citizens die off the country will become more liberal. Others argue that young people will become conservative as they age, maintaining the status quo. Do we become more conservative as we age? There are studies on both sides of the issue as evidenced in this 2012 article and this 2014 article.

I won’t attempt to resolve this issue here but, instead, use it as a backdrop for another interesting finding in my demographic study of IMDB’s Top 250 Movies. Age does appear to factor in IMDB voting. Take a look at these results by Age Group:

Avg. IMDB Rating
Age Group All Male Female
Under 18             8.7             8.8                       8.6
18 to 29             8.5             8.5                       8.4
30 to 45             8.3             8.4                       8.3
Over 45             8.2             8.2                       8.0

As IMDB voters get older, the average rating for the same group of movies is lower. It doesn’t matter whether the groups are male or female. The pattern is still the same. The fact that the avg. ratings for the female groups is consistently lower than the male groups is probably due to the bias towards male-oriented movies in the Top 250. Is this further evidence that we get more conservative as we get older?

I’ll offer up a counter-argument, maybe we get smarter as we get older. There are scientific studies that support this including those cited in this 2011 article. There is some IMDB support for this argument, as well. One of the demographic groups that IMDB captures data for is the Top 1,000 IMDB voters. These are voters who have rated the most movies on IMDB and presumably have watched the most movies. The avg. IMDB rating from this group for the Top 250 Movies is 7.9. Perhaps, the more movies that you watch, the smarter you get at differentiating one movie from another. If so, then maybe the lower average ratings for the older age groups are more representative of the experience gained from watching a greater number of movies. Whether we get more conservative or smarter as we age, it would be wise for the older moviegoer to recognize that the avg. IMDB rating is heavily influenced by males aged 18 to 29. You’ll need to apply a Senior Discount to the rating. What do you think?


Screening Room and the Long Shadow of Netflix

Over the last few weeks, an interesting news item that might significantly impact the future of home entertainment has been making its way into the mainstream media. A new home video service called Screening Room is being pitched in Hollywood. Here is the idea. For $150, customers would buy a set-top box, probably similar to a cable TV box, which would enable them to rent, for 48 hours, a movie on the day it is released in the movie theaters for $50 a movie, with a share of that $50 going back to movie theaters. But wait, the renter of the movie would also get two tickets to see the same movie at the theaters. This is either a crazy idea or the next big thing on the horizon for an industry already in great flux.

My initial reaction was that this is a crazy idea because the $50 cost is out of reach for the average moviegoer. What gives me pause, though, is that the idea belongs to Sean Parker, the co-founder of Napster and former president of Facebook. For moviegoers, he was the character played by Justin Timberlake in The Social Network. Screening Room is backed by a powerful cadre of a who’s who in Hollywood, including producer Brian Grazer and directors Steven Spielberg, Ron Howard, J.J. Abrams and Peter Jackson. Next week it is being demonstrated to industry professionals at the CinemaCon convention in Las Vegas. Stay tuned for further developments. This story could have legs.

For now, let’s consider, is the idea viable and why are powerful moviemakers getting behind it? Many home entertainment customers have bought a streaming device, Roku, Apple TV etc., for $50 – $100. The $150 for the set-top box probably isn’t unreasonable if there is sufficient demand for $50 movies. That’s the big question. Event movies, like Star Wars: The Force Awakens, that are candidates for viewing parties, would be a reasonable target for the service. It would be an extension of the demand that exists today for pay per view boxing matches. Families, that often go to the movies as a family, are another potential market. Couples with young children might also be a target when you add to the movie theater costs the cost of hiring a babysitter. But, for Singles, and Couples with no children, the economic cost benefit doesn’t seem to make sense.

As for moviemakers, the motivations might be complex. There is the possibility that it could add to the box office. For example, young couples with kids who would rather stay home than pay for a sitter, might see more first run movies than they normally would go to the theater to see. The biggest motivation for the movie industry, though, may be the looming threat posed by Netflix and other streaming services.

When Netflix streamed Beasts of No Nation (Idris Elba was a Screen Actors Guild Award winner for the film.) for all of its subscribers on the same day it was released in movie theaters, it sent a shot across the bow of the movie industry, particularly theater owners. The four major theater chains in the US boycotted the film and it ended up getting only a limited release in 200-250 independent and art film theaters. Another Sundance favorite, The Fundamentals of Caring with Paul Rudd and Selena Gomez, was purchased by Netflix and will be streamed on June 24, 2016 on Netflix. It is dawning on the industry that moviegoers may not need to actually go to movie theaters to see first run movies. They can be delivered to them at home.

Screening Room may be seen by the movie theater industry as a financially viable alternative to Netflix, at least until Screening Room also eliminates their payoff to the middle man, making it more affordable to average movie watchers. After all, that’s how Netflix does it.

What do you think? Is it a viable concept?

Why Don’t More Women Rate Movies on IMDB?

Last October Meryl Streep was at the London Film Festival promoting her new movie Suffragette, a film about the struggle to secure the vote for women in Britain. She used the opportunity to criticize Rotten Tomatoes for its underrepresentation of female critics in its Tomatometer, the tool Rotten Tomatoes uses to grade movies. She pointed out that there were 168 women to 760 men among the critics used by Rotten Tomatoes. She felt that this one sided bias was negatively impacting women-driven movies at the box office. In an interview with the Daily Beast she said “I submit to you that men and women are not the same, they like different things. Sometimes they like the same thing but sometimes their tastes diverge. If the Tomatometer is slighted so completely to one set of tastes that drives box office in the United States.”

Meryl Streep is spot on when she says that sometimes men and women like the same movies but often their tastes differ. For example, in a recent demographic study I put together of the IMDB Top 250 Movies, here were the top 5 movies for men and women:

Men Women
The Shawshank Redemption The Shawshank Redemption
The Godfather Schindler’s List
The Godfather: Part II The Godfather
The Dark Knight 12 Angry Men
The Good, the Bad and the Ugly The Lord of the Rings: The Return of the King

Men and women agree on two of the top 5 but disagree on the other 3.

Is Rotten Tomatoes restricting, or not encouraging, women to participate as critics for their website, or are women simply less interested in film criticism? According to the Motion Picture Association of America (MPAA), Females represented 52% of all moviegoers in 2014. In my study of IMDB’s Top 250, only 16% of the votes cast for Top 250 movies were by women, interestingly close to the 18% participation of female critics on Rotten Tomatoes. Here are the female participation rates by age groups for the Top 250 study:

Age Group Female % of IMDB Votes
Under 18 21%
18 to 29 18%
30 to 45 14%
Over 45 16%

Some hypotheses for these results might include:

  1. Women are not as comfortable with technology as men, particularly in the older age groups.
  2. There is an unequal sharing of household responsibilities, particularly during the 30 to 45 child rearing years.
  3. Women go to the movies more than men because they don’t have time to watch movies when they are home. Men, on the other hand, find time to watch movies at home.
  4. There is a male bias to the Top 250 movies list.
  5. There are other things women would rather do than rate movies.

The data above is consistent with the first two hypotheses. The third hypothesis may also be a factor. My Top 250 data demonstrates that women have higher IMDB participation rates for newer movies, which supports greater viewership at movie theaters than at home.  When ranked by release date from the oldest to the newest and divided into two groups of 125, there is clearly greater participation across all age groups for newer movies.

Age Group Oldest 125 Movies (Median Release Date 1963) Newest 125 Movies (Median Release Date 2003)
Under 18 16% 23%
18 to 29 15% 20%
30 to 45 12% 15%
Over 45 16% 17%
All Ages 14% 18%

I think you see a convergence in the participation rates as the age groups get older because the old movies become more contemporary as the groups get older. But even for the oldest group, who may have watched many of the old movies when they were new, the participation is greater for the newer movies.

The fourth hypothesis is certainly true. In the Top 250 there are 142 movies that men rate higher than women and only 66 that women rate higher than men. But, in those 66 movies that women rate higher than men, women still are only 16% of the total vote.

In the final analysis, though, when you control for the first four hypotheses, I can’t get female participation in IMDB voting to a level greater than 23%. In fact, the single movie in the Top 250 with the highest female participation, the Audrey Hepburn classic Roman Holiday, has a participation rate of only 37%. By the way, the more contemporary version of the same movie, Notting Hill, has only 38% female participation.

All of which leaves me with hypothesis 5, rating movies is one of the things that you do on Mars rather than on Venus. Whatever the reason, Meryl Streep’s concern is real and change is hard. It happens one IMDB vote at a time.

Have I struck a nerve? Do you have any other hypotheses? Please leave comments.

Post Navigation