Beam is a job orchestration and data processing framework from Apache foundation, from a bit of reading, it seems similar to Airflow.
Looking for any good book recommendation on Apache beam Python SDK. I know I can read the documentation, but it seems it is quite scattered and involves a lot of navigation to build a foundation or mental model.
Some books that I did come across, focus on the Java SDK which is not what I want. Hence wondering if anyone can recommend any book focussing on the python API.
Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing
This is the only good book that will allow you to understand how Beam works. It’s written by one of the engineers working on Beam.
Also, the Python SDK is not great if you are looking for pipelines that scale well and are going to work with non-google sources and sinks, like Kafka, PG, Clickhouse. We tried it but it’s expensive to run and not very reliable. Haven’t tried the Java SDK tho… Maybe it’s better.
At the company I work for, we switched to Apache Flink on Java. Works better and is very reliable and consistent.