🧵We are developing an #AI to analyse #NBA data. It involves one #LLM that generates a SQL query on a database, and another to transform that data into a chart and also provide a brief text summary. Here are some challenges that we've faced. #Developers #SoftwareEngineering #SportsAnalytics - ThreadSky

5thqsports.bsky.social • 13 days ago

🧵We are developing an #AI to analyse #NBA data. It involves one #LLM that generates a SQL query on a database, and another to transform that data into a chart and also provide a brief text summary. Here are some challenges that we've faced.

#Developers #SoftwareEngineering
#SportsAnalytics

Comments

5thqsports.bsky.social•13 days ago

Some LLMs are great at different things depending on their training data. For example Qwen2.5-coder generates Apache ECharts input from raw data quite well, but other LLMs available through Ollama are less good. Have not tested other libs like Chart.js or D3 or VegaLite.

5thqsports.bsky.social•13 days ago

Further, some LLMs might seem like they'd be a good choice but fall over when faced with complex instructions. We are using DuckDB, and they provide a text to SQL model! Works well for simple questions but fails fast for queries that have a handful of equations in the SELECT list or multiple filters

5thqsports.bsky.social•13 days ago

We settled on Athene-v2 for now. It is able to digest a schema, has some implicit knowledge about basketball, and can create relatively complex queries.

Contrast this to Qwen2.5-coder. It has less implicit basketball knowledge and struggles to come up with eFG% queries.

5thqsports.bsky.social•13 days ago

Finally, schema matter the most. Many of these models fall over with 3-4 way joins, and create garbage responses. Highly denormalized schemas work the best. Instead of having a dense column of 'description's, we instead have many columns like 'is_field_goal', 'is_personal_foul', etc.

5thqsports.bsky.social•13 days ago

I like to think of this as a row to column transposition creating a sparse matrix.

Naming consistency also matters. If some columns are pluralized and some aren't the LLM is very likely to confuse them and generate an incorrect column name.

5thqsports.bsky.social•13 days ago

Additionally, it's worth having the LLM try again. If a query fails to run, capture the error message and have an "error fallback" prompt. This plays well with LangChain, and trying 2-3 times seems to resolve the issues for us.

Comments

Posting Rules

Reply