Does anybody know the "VertiPaq" backstory (later renamed xVelocity)? It's fascinating how early Microsoft had a columnar, in-memory engine (2006) that integrated into all their tools (Excel, PowerPivot, SSAS, PowerBI) to this day. But yet, there is almost no information available. Why? - ThreadSky

ssp.sh • 34 days ago

Does anybody know the "VertiPaq" backstory (later renamed xVelocity)? It's fascinating how early Microsoft had a columnar, in-memory engine (2006) that integrated into all their tools (Excel, PowerPivot, SSAS, PowerBI) to this day.

But yet, there is almost no information available. Why?

Comments

alexondata.bsky.social•33 days ago

Another thing to note is that VertiPaq handles joins through pre-computed hashmaps (called relationships) that are rebuilt anytime a table in the join changes. The hashmap sits in the CPU cache and this avoids regular join perf concerns. Another read-focused design choice.

ssp.sh•33 days ago

Thanks for sharing, that is super interesting!!

alexondata.bsky.social•33 days ago

Forgot to point out this doesn’t apply for CCI in SQL Server or Synapse, just the read heavy analytics focused implementations of VertiPaq.

ssp.sh•33 days ago

Thank you Alex. Is it ok to add to my note? I put it as a quote: https://www.ssp.sh/brain/vertipaq/#how-does-it-work

alexondata.bsky.social•33 days ago

Sure thing! I’ll DM you some technical book references shortly.

alexondata.bsky.social•33 days ago

Actually I’ll just put them here.
Alberto Ferrari & Marco Russo (2024) Optimizing DAX. SQLBI Corp. Chapter 11-12.

ssp.sh•34 days ago

#datasky, anyone?

ssp.sh•34 days ago

@marcorusso.bsky.social 🤔 Your articles are the first ones that show up, either on YouTube or Blog.

https://youtu.be/85rJ-9vQBbU?si=YCIiF5KR5cGA527i

ssp.sh•34 days ago

Collected all feedback and notes on https://www.ssp.sh/brain/vertipaq/.

dbrtly.bsky.social•34 days ago

Look up Jeffery Wang. He was heavily involved.

rifkiamil.bsky.social•34 days ago

@ssp.sh Here is the 1st online demo Project Gemini, which became PowerPivot with VertiPaq https://youtu.be/KrpBLFRQFnk?si=bKi8KhQZZIt_LfHW&t=87, I saw at the time as Microsoft Office Advisory Council or M$ TAP programs. Even V-Order in Fabric is Microsoft's special way of enhancing performance from Delta Lake.

asliceoftom.bsky.social•34 days ago

Thanks for helping out @rifkiamil.bsky.social !

ssp.sh•34 days ago

Interesting, I didn't know it was called "Project Gemini". Thanks for the share. It is so fun to watch 15-year-old YouTube; it shows me how old I've gotten 😀

rifkiamil.bsky.social•34 days ago

One more project name; I can't recall it right now, but I will find it.

rifkiamil.bsky.social•34 days ago

It is also integrated into Azure Synapse Dedicated SQL (formerly SQL DW) when setting the table as a Columnstore index. Just make sure you know about the 8-byte rule. Some good history and knowledge here https://www.youtube.com/watch?v=e_VPylrmcXw & https://www.youtube.com/watch?v=oWkcNidQaXw

joschkos.bsky.social•34 days ago

Here at 15:00 Amir talks about the In Memory needs at Microsoft.

https://youtu.be/izOppfQGanI?si=KBxsoe3gwRtpw4uV

ssp.sh•34 days ago

Thanks so much for sharing. This helps a lot.

I added some of this conversation to my note, trying to decipher the history. I'll expand along as I learn.

https://www.ssp.sh/brain/vertipaq/

joschkos.bsky.social•34 days ago

This is cool. Very interesting.
It is Cristian Petculescu.

runnin-l8.bsky.social•34 days ago

You might want to see if Marco Russo or Alberto Ferrari from https://sqlbi.com know. Also @grrlgeek.bsky.social may have some insight.

ssp.sh•34 days ago

Thanks so much; Marco's articles came up first, and I tagged him, but I am not sure if he is active here :)

Looking forward to Jes's response if she knows more.

grrlgeek.bsky.social•34 days ago

I remember when it was new and NDA! What do you want to know about it?

ssp.sh•34 days ago

How does it work? how does it compare to today's columnar in-memory formats such as Apache Arrow? how it has evolved over the years? How does it integrate into all the different Microsoft services (SSAS, Excel, Power BI, etc.)? Things like this :)

ssp.sh•34 days ago

Alberto replied on LinkedIn: https://www.linkedin.com/feed/update/urn:li:activity:7289984556433121280?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7289984556433121280%2C7289987366843240448%29&replyUrn=urn%3Ali%3Acomment%3A%28activity%3A7289984556433121280%2C7290014684139646977%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287289987366843240448%2Curn%3Ali%3Aactivity%3A7289984556433121280%29&dashReplyUrn=urn%3Ali%3Afsd_comment%3A%287290014684139646977%2Curn%3Ali%3Aactivity%3A7289984556433121280%29

And Marco on Twitter: https://twitter.com/marcorus/status/1884254005876158521

It seems there is no public information available by Microsoft, only in discussion or when asking certain people directly.

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service