Since this is a technology-centric blog entry, let's get back to the topic at hand. I get excited about fantasy football every fall, and the pinnacle of any fantasy sports league is the player draft. I have also been fascinated by how good these LLMs are at meeting my fantasy sports needs. As someone who has played fantasy sports since 1996, I have trusted sources. For this blog entry, I am going to use the rankings from Yahoo! Fantasy Sports. This is what I did.
Purpose: Leveraging ChatGPT, Google Gemini Advanced, Meta AI and MSN Copilot chatbots, I asked them to give the top 50 quarterbacks for this season.
Scope: I focused on NFL quarterbacks because only 32 quarterbacks are playing any given week. Unlike kickers, quarterbacks have a significant impact on a football game.
Language Learning Models Used:
| AI Vendor | LLM |
|---|---|
| OpenAI Chat GPT | GPT-4o |
| Google Gemini | Gemini 1.5 Pro |
| Meta AI | Llama 3. |
| MSN CoPilot | Unknown |
The prompt I used for the four sites was:
"Give me the list of the top 50 quarterbacks for my fantasy American football league. Here is the scoring key:
Offense League Value (points)
Passing Yards: 25 yards per point; 5 points at 350 yards
Passing Touchdowns: 6
Interceptions: -1
Rushing Yards: 10 yards per point; 5 points at 200 yards
Rushing Touchdowns: 6
Receptions: 1
Receiving Yards: 10 yards per point; 5 points at 200 yards
Receiving Touchdowns: 6
Return Touchdowns: 6
2-Point Conversions: 2
Fumbles Lost: -2
Offensive Fumble Return TD: 6"
I compared the LLM results with Yahoo! Fantasy Sports' top 50 quarterbacks (QB). As of August 16th, 2024, here is the list:
- Josh Allen - Buffalo Bills
- Jalen Hurts - Philadelphia Eagles
- Lamar Jackson - Baltimore Ravens
- Patrick Mahomes - Kansas City Chiefs
- Anthony Richardson - Indianapolis Colts
- C.J. Stroud - Houston Texans
- Kyle Murray - Arizona Cardinals
- Joe Burrow - Cincinnati Bengals
- Dak Prescott - Dallas Cowboys
- Jordan Love - Greenbay Packers
- Jayden Daniels - Washington Commanders (rookie)
NOTE: The rest of the list is here: Top 50 QBs - 8/16/2024 - Google Sheets
Here is how the various LLM models ranked these quarterbacks.
| Yahoo! Rank | Name | ChatGPT | Meta AI | MSN Copilot | Google Gemini |
|---|---|---|---|---|---|
| 1 | Josh Allen | 1 | 2 | 2 | 1 |
| 2 | Jalen Hurts | 2 | 4 | 4 | 2 |
| 3 | Lamar Jackson | 4 | 3 | 3 | 4 |
| 4 | Patrick Mahomes | 3 | 1 | 1 | 3 |
| 5 | Trent Richardson | 5 | 5 | 5 | 19 |
| 6 | C.J. Stroud | 6 | 6 | 6 | 28 |
| 7 | Kyle Murray | 8 | 8 | 8 | 10 |
| 8 | Joe Burrow | 7 | 7 | 7 | 6 |
| 9 | Dak Prescott | 9 | 10 | 10 | 14 |
| 10 | Jordan Love | 11 | 9 | 9 | 21 |
| 11 | Jayden Daniels | 17 | 11 | 11 | NOT LISTED |
It appears Google Gemini's results (https://gemini.google.com/app/1ab3b15202d9e4f1) did not incorporate the 2024 NFL Draft or the changes in free agency.
Unlike Google Gemini, ChatGPT, Meta AI, and MSN, Copilot aggregated ranking from other sites. These sources are listed in the various result sets. Here are the links to the prompts and the appropriate result sets:
The three LLMs didn't do number crunching to generate the results but instead pulled data from other websites and aggregated the data.
Since I was doing a rudimentary analysis of the various LLMs to meet my needs with my upcoming Fantasy Football draft, I didn't write any code but simply leveraged existing algorithms in Microsoft Excel.
Results:
Here is the summary of the results:
| No. | Player | Teams | Depth | Yahoo! Sports | AI Rank |
|---|---|---|---|---|---|
| 1 | Josh Allen | Buf | 1 | 1 | 2 |
| 2 | Jalen Hurts | Phi | 1 | 2 | 4 |
| 3 | Lamar Jackson | Bal | 1 | 3 | 3 |
| 4 | Patrick Mahomes | KC | 1 | 4 | 1 |
| 5 | Anthony Richardson | Ind | 1 | 5 | 5 |
| 6 | C.J. Stroud | Hou | 1 | 6 | 6 |
| 7 | Kyle Murray | Ari | 1 | 7 | 8 |
| 8 | Joe Burrow | Cin | 1 | 8 | 7 |
| 9 | Dak Prescott | Dal | 1 | 9 | 10 |
| 10 | Jordan Love | GB | 1 | 10 | 9 |
| 11 | Jayden Daniels | Was | 1 | 11 | 11 |
| 12 | Brock Purdy | SF | 1 | 12 | 12 |
| 13 | Jared Goff | Det | 1 | 13 | 15 |
| 14 | Tua Tagovailoa | Mia | 1 | 14 | 13 |
| 15 | Caleb Williams | Chi | 1 | 15 | 16 |
| 16 | Trevor Lawrence | Jax | 1 | 16 | 14 |
| 17 | Kirk Cousins | Atl | 1 | 17 | 17 |
| 18 | Matthew Stafford | LAR | 1 | 18 | 19 |
| 19 | Justin Herbert | LAC | 1 | 19 | 18 |
| 20 | Geno Smith | Sea | 1 | 20 | 22 |
| 21 | Aaron Rodgers | NYJ | 1 | 21 | 23 |
| 22 | Deshaun Watson | Cle | 1 | 22 | 20 |
| 23 | Baker Mayfield | TB | 1 | 23 | 21 |
| 24 | Will Levis | Ten | 1 | 24 | 24 |
| 25 | Derek Carr | NO | 1 | 25 | 26 |
| 26 | Taysom Hill | NO | 26 | ||
| 27 | Daniel Jones | NYG | 1 | 27 | 25 |
| 28 | Bryce Young | Car | 1 | 28 | 28 |
| 29 | Bo Nix | Den | 1 | 29 | 30 |
| 30 | Justin Fields | Pit | 2 | 30 | 27 |
| 31 | Russell Wilson | Pit | 1 | 31 | 29 |
| 32 | JJ McCarthy | Min | 1 | 32 | 35.5 |
| 33 | Jacoby Brissett | NE | 33 | 33 | |
| 34 | Sam Darnold | Min | 34 | 32 | |
| 35 | Gardner Minshew II | LV | 1 | 35 | 34 |
| 36 | Drake Maye | NE | 1 | 36 | 31 |
Regarding the AI Rank, I removed the Google Gemini results using the Standard Deviation algorithm STDEV and the Variance algorithm VAR since they were outliers compared to the other results. Here is the link to the spreadsheet (ai-fantasyqb.xlsx - Google Sheets)
Conclusion:
I didn't expect the diversity of the results from the four AI engines and how these AI engines leverage third-party content from other respectable websites. The other thing I didn't expect is that the results generated from their APIs are quite different since they don't pull data from third-party websites. In summary, I envision using something other than AI and the various respective LLMs to address my fantasy sports needs. I expect sports vendors like ESPN, Yahoo!, and sporting organizations to probably have sophisticated AI, LLMs, and machine learning technologies, which will be more reliable for at least the next couple of years.
On a personal note, if you are an aspiring data scientist, don't be discouraged because you don't have a programming background or understand various algorithms like backward propagation neural networks. You should be curious to learn from the data.
No comments:
Post a Comment