A modular rewrite featuring numerous enhancements and bug fixes

[9ao9ai9ar]

The only downsides of my rewrite that I can think of are the higher memory usage (by about 50%) and increased dependencies.

[mhdadk]

Hey @9ao9ai9ar, thank you very much for this PR! It seems like you really cleaned up the code. Adding tests was an especially nice touch.

Before merging this PR, because it is quite large (49 files changed), I'd like to review it in more detail to make sure it is OK. Unfortunately, I'm not sure when I will have the time in the near future to do this. I'm going to keep this PR open until I do so.

Nevertheless, I wanted to write to you to give you some positive feedback on your work.

[9ao9ai9ar]

Thanks for your words of encouragement. I will be pushing out minor commits in the meantime, like updating the README and upgrading dependencies (Pydantic v2.9.0 is just around the corner, promising to fix the high memory consumption).

I do have an idea to improve the output format, but I will withhold work on that until next year. The idea is to first move all non-body fields of the posts into the (non-standard but widely supported) YAML metadata section at the beginning of the Markdown files. Secondly, individual answers should probably be separated into individual files to keep them as close to the original as possible without modifying the Markdown content by inserting the section markers. Lastly, the licenses of the posts (currently there are 3 versions of CC in use) should be included and linked. I can open an issue if you're interested in further discussion.

[mhdadk]

All these changes sounds good to me. I can have a look at them as well when you push them (and when I have some more time).

[9ao9ai9ar]

I have three news to share with you, one good, one bad, and one neutral:

1. I've thought of another important feature to add to this project: include an option to download all images as files or base64 encoded data blobs. In the data dumps, only the image URLs are saved, so there is still a risk of URL rot and losing the pictures. I can probably start the work in the second half of the year.
2. The "slight" increase in dependencies has negatively affected the ability to run the script on some platforms. For example, I wasn't able to get the project to build on NetBSD due to compile errors in PyO3. Pydantic 2.9 also didn't fix the high memory consumption; they are tackling the memory issues again in Pydantic 2.11, but if no progress were made either in this release, I might have to reconsider the choice of incorporating this library in the code, which would be a shame as I kind of like their API.
3. Since this is more a rewrite than a refactor, it's less important to do piecemeal commits in my humble opinion, so as I intend to do a git rebase to squash the commits into one, I will mark this pull request as draft for the time being. In the future, I will open feature branches to work on and do the rebases there.

[mhdadk]

I have three news to share with you, one good, one bad, and one neutral:

1. I've thought of another important feature to add to this project: include an option to download all images as files or base64 encoded data blobs. In the data dumps, only the image URLs are saved, so there is still a risk of URL rot and losing the pictures. I can probably start the work in the second half of the year.

2. The "slight" increase in dependencies has negatively affected the ability to run the script on some platforms. For example, I wasn't able to get the project to build on NetBSD due to compile errors in PyO3. Pydantic 2.9 also didn't fix the high memory consumption; they are tackling the memory issues again in Pydantic 2.11, but if no progress were made either in this release, I might have to reconsider the choice of incorporating this library in the code, which would be a shame as I kind of like their API.

3. Since this is more a rewrite than a refactor, it's less important to do piecemeal commits in my humble opinion, so as I intend to do a git rebase to squash the commits into one, I will mark this pull request as draft for the time being. In the future, I will open feature branches to work on and do the rebases there.

Nice work!