Brown, M. A., Gruen, A., Maldoff, G., Messing, S., Sanderson, Z., & Zimmer, M. (2024). Web Scraping for Research: Legal, Ethical, Institutional, and Scientific Considerations (arXiv:2410.23432). arXiv. https://doi.org/10.48550/arXiv.2410.23432
A nice overview of the different considerations, with recommendations, for researchers who may want to use web scraping.
As scraping for large-scale AI training becomes more common, scraping for other research purposes may become more challenging.
@mae
@chadkoh a remarkably nihilist conclusion! Do I understand the authors correctly that they believe it is impossible to predict what effect protocol design decisions will have on the world, therefore ... we shouldn't try? Or, it would even be harmful if we tried?