Jump to content


Photo

Selenium For Automating Nexus Downloading


  • Please log in to reply
2 replies to this topic

#1 Snowden42

Snowden42

    Prisoner

  • Members
  • 7 posts

Posted 01 June 2017 - 05:56 PM

Hi Folks,

 

I've been playing around with using Selenium for automating the boring task of downloading files from the Nexus. The nice thing about Selenium is it doesn't really circumvent the Nexus, since it's literally just a macro system for controlling your browser. I'd need to check the ToS to make sure its kosher though.

 

Would this be wanted? Selenium doesn't capture any information from your browser specifically, and it does have dependencies--The only browser I'm sure this would work with is Firefox. Unfortunately this does mean it can't monitor downloads. So the only way to have it fully automated is for it to just download all files in one go.

 

I'm thinking you'd probably just package a JSON file with a mapping of "LinkName":"URL", and it would automate opening each link, going through the necessary menus, and downloading them.

 

Also I should note, in my limited testing, it seems like the Selenium WebDriver for Firefox doesn't reuse existing cookies. This means that you'd be required to pass your Nexus creds in plaintext to the utility in order for it to login on your behalf.

 

Thoughts, suggestions?


Edited by Snowden42, 01 June 2017 - 05:58 PM.

  • 0

#2 airbreather

airbreather

    Guard

  • Developers
  • PipPip
  • 114 posts

Posted 02 June 2017 - 07:33 AM

For reference, here's the part on the Automation Survey wiki page that discusses the problem with relevant links and my interpretation of the furthest we can go politely with the tools that Nexus gives us (or at least, with the tools that existed when I wrote that).

 

Personally, I don't see a Selenium-driven process being materially different from making a bunch of nxm:// links and opening all of them with Mod Organizer (in fact, doing the nxm:// idea might even be preferable from the Nexus's perspective because then their servers don't have to send you as much static overhead).

 

The reason why I don't see it being actually different is that the user isn't actually doing the organic website interaction that the site operators want you to be doing to download their files (which includes the possibility of ad clicks, throws the endorse buttons and related mod links in front of them as they download, makes them feel the pain of downloading huge amounts of content from the pleb non-Premium servers, etc.).  It just, for the most part, looks like the same thing to the Nexus, though it would stick out like a sore thumb if someone actually looked at the server logs.

 

I think a polite solution to the "too many clicks" downloading problem really needs to come after serious help from the Nexus itself, or mod authors putting their stuff on other hosts who are more friendly towards automation.  MontyMM's "mod collections" suggestion at that link above sounds like a really good solution (especially if you can say "premium users can download the whole thing with one click").

 

This is all just my opinion, though.


  • 0

#3 Greg

Greg

    High King

  • Super Moderators
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,460 posts

Posted 02 June 2017 - 09:27 PM

Selenium was originally developed as a tool to automated web UI testing but has certainly grown in its capabilities over the years. While Selenium could certainly be used to automate downloading all the necessary mods, I don't think it would be a good idea because it would hammer Nexus' servers with excessive traffic and DarkOne may consider such a use a violation of the terms of service.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users