Category: Delta Lake

There is a very heavy penalty to pay in Delta Lake when transactions conflict. This note explores how the delta format achieves isolation between transactions. In the best case scenario we enjoy lock free concurrency, in the worst case we waste compute and storage. That is the consequence of how concurrency is achieved with the…

I wanted to clarify to myself how we achieve Atomicity in Delta Lake, and it’s easier for me to compare it to traditional databases. Atomicity means that when we commit a transaction, it’s an all-or-nothing operation. Either the transaction succeeds or it fails, nothing in between. Consistency means that despite our transaction being atomic, if…