jared252016

joined 11 months ago
 

I love Aria2, but I'm building a web scraper / crawler and I need to download hundreds of thousands of files. Aria2 locks up around the 20,000 file mark. Is there another download manager that could possibly be able to achieve what I'm trying to do? or a more recent fork of Aria2?

I have a workaround I believe, which is to use the API to determine how many files are in queue and sleep indefinitely until there is < 1000, but I'm not sure this is the most effective. It kind of significantly slows down the download pipe.

The issue seems to lie with connections timing out in aria2, which cause them to get locked up and they have to be manually cleared. I have my timeout set at 10 seconds, but that doesn't seem to matter. I've considered running a schedule task to clean them up, but was going to give downloading with Python a try first.

Any suggestions would be appreciated.

[–] [email protected] 1 points 10 months ago (1 children)

ThePirateBay, the most notorious site in the world, uses Cloudflare. This isn't China. Wiretapping is illegal in most circumstances, and that's essentially what it would be doing.

[–] [email protected] 1 points 10 months ago

Not all docker-compose files are development, although it is in this case. The documentation, if available, will tell you what each env var does.

You know the docker-compose.yml will have everything you need though, you just need to tweak it. Absolutely nothing wrong with seeking out the docker-compose as most env variables are self explanatory.

[–] [email protected] 0 points 10 months ago (2 children)

Did you set the environment variables for the database and what not?

When you get stuck like this, always look at the docker-compose.yml file if provided. You can find this one here: https://github.com/mealie-recipes/mealie/blob/mealie-next/docker/docker-compose.yml

It lists all the environment variables that need to be set.

Alternatively you should be able to use docker compose in WSL, it'll still show up in Docker Desktop.

[–] [email protected] 1 points 11 months ago

I would do the ethical thing and explain the privacy implications of you hosting it but him hosting it too. Honestly I'd turn on end to end encryption.

You having their data and him having his family's data is a big responsibility. You need adequate security both online and physically.

There's also the fact that he has access to it if he's an admin. Not everyone can handle that responsibility.

As to your original question, Nextcloud breaks often, relative to how often the server will have problems. You'd def need SSH access.

It would probably be better for you to get an ASUSTOR NAS in terms of hardware. It Supports apps and Nextcloud is one of them.

It also has support other than you, which depending on a few factors (magic 8 ball), might be more time consuming than you'd think. They may not want to deal with hardware either if something did occur.

[–] [email protected] 1 points 11 months ago

From your diagram you should be able to access it already. It's what is called dual NAT where a router is behind a router.

Otherwise you could turn off DHCP on one and use static IPs / routes

Would it be possible to plug your media server into both routers?