Jump to content


Photo

Skyrim Nexus Downloader


  • Please log in to reply
40 replies to this topic

#1 d2allgr

d2allgr

    Prisoner

  • Members
  • 9 posts

Posted 23 January 2013 - 12:47 PM

I know that Thunderbolt and/or related tools are in the backburner as z929669 mentioned (until you manage to tie up the in-progress release) but would something like this be of any use?
Sample output so far:
Logged in to Skyrim Nexus
STEP - Skyrim Total Enhancement Project
Id: 29961, Name: STEP v2-0-1a, Filename: STEP v2-0-1a-11.rar
Url: http://skyrim.nexusmods.com/Libs/Common/Managers/Downloads?Download&nid=110&fid=29961
Id: 59027, Name: STEP v2-1-0c, Filename: STEP v2-1-0c-11.rar
Url: http://skyrim.nexusmods.com/Libs/Common/Managers/Downloads?Download&nid=110&fid=59027
Id: 59738, Name: STEP v2-1-1b, Filename: STEP v2-1-1b-11-2-1-1b.rar
Url: http://skyrim.nexusmods.com/Libs/Common/Managers/Downloads?Download&nid=110&fid=59738
Id: 84733, Name: STEP v2_2_0, Filename: STEP v2_2-11-2-2.7z
Url: http://skyrim.nexusmods.com/Libs/Common/Managers/Downloads?Download&nid=110&fid=84733
Id: 90337, Name: STEP v2_2_0a, Filename: STEP v2_2_0a-11-2-2-0a.7z
Url: http://skyrim.nexusmods.com/Libs/Common/Managers/Downloads?Download&nid=110&fid=90337
Logged out
Press any key to continue . . .


So far (after messing with parts of the Nexus Mod Manager code) it can login to the nexus server, find a mod based on its id, display all the public uploaded files (name, filename and id) and grab a premium (or not) download url.

Would something like this be of any interest (to build upon if need be) so that I should contact DarkOne first to check if I need permission for sth like this.

My purpose was to check how difficult it would be, without directly opening the page and messing with cookies and such, using the same interface that the mod manager uses (making it official sort of?) and the result is this, using 30 of the 2500+ files from the mod manager source code and 50 lines of code to tie it all together.

Let me know if it will be any useĀ pursuingĀ this (or if not, in case behind the scenes you already have this rdy)
  • 0

#2 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,258 posts

Posted 23 January 2013 - 01:20 PM

We do have some working XP with the Nexus metadata and a basic API to retrieve this information using Python; however, I am sure that whatever ideas that you have might be of interest as well. We could certainly use some help developing an application or a number of other tasks. If you are interested in helping out, we will probably begin tackling things after this next release, so let us know if you are interested and what you are interested in doing. --same goes to anyone else with regard to site development, admin and maintenance as well as application dev.

#3 stoppingby4now

stoppingby4now

    Sleepy

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,281 posts

Posted 23 January 2013 - 02:22 PM

We should talk to Tannin to get some insight as to what he did. IIRC, he is using actual NMM source to handle downloads. It may be the way to go in order to play it "safe".

#4 Daagar

Daagar

    Prisoner

  • Members
  • 48 posts

Posted 25 January 2013 - 01:10 PM

One area where the existing tools "fail" is handling the massive amount of timeouts/503s/etc errors that the nexus likes to throw. One nice improvement in any new attempt at an 'auto-downloader' would be some graceful way of handling this (assuming the intention is to handle downloading, not just URL gathering).
  • 0

#5 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 12,107 posts

Posted 25 January 2013 - 02:03 PM

One area where the existing tools "fail" is handling the massive amount of timeouts/503s/etc errors that the nexus likes to throw. One nice improvement in any new attempt at an 'auto-downloader' would be some graceful way of handling this (assuming the intention is to handle downloading' date=' not just URL gathering).

On this note, a simple "retry loop" every 30sec to 1min on failed attempts would handle this perfectly.

#6 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 25 January 2013 - 02:47 PM

One area where the existing tools "fail" is handling the massive amount of timeouts/503s/etc errors that the nexus likes to throw. One nice improvement in any new attempt at an 'auto-downloader' would be some graceful way of handling this (assuming the intention is to handle downloading' date=' not just URL gathering).

I've just encountered this again, as I had to redownload all of STEP from scratch.  Doing it with the usual tools is no fun at all.  I did a bit of work on this months ago, but I lost the old data and started again.  The open source download tool Aria2 does an excellent job of dealing with poor behavior of the Nexus servers.  I've written a script that processes a download list of file ID's, and then uses all the available Nexus servers as file mirrors.  Aria is very good at intelligently spreading the downloads over the mirrors, and retrying alternatives when disconnected.

The only problem is, it might be a bit too efficient for the liking of the Nexus (though it doesn't seem unreasonable to me, since the same amount of data is being downloaded and the demand spread across the available servers - it only makes one connection to each server.)  I'll have to speak to the admins before unleashing a possible Nexus-taunting device on the STEP forum.
  • 0

#7 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,258 posts

Posted 25 January 2013 - 03:51 PM

One area where the existing tools "fail" is handling the massive amount of timeouts/503s/etc errors that the nexus likes to throw. One nice improvement in any new attempt at an 'auto-downloader' would be some graceful way of handling this (assuming the intention is to handle downloading' date=' not just URL gathering).[/quote']
I've just encountered this again, as I had to redownload all of STEP from scratch.  Doing it with the usual tools is no fun at all.  I did a bit of work on this months ago, but I lost the old data and started again.  The open source download tool Aria2 does an excellent job of dealing with poor behavior of the Nexus servers.  I've written a script that processes a download list of file ID's, and then uses all the available Nexus servers as file mirrors.  Aria is very good at intelligently spreading the downloads over the mirrors, and retrying alternatives when disconnected.

The only problem is, it might be a bit too efficient for the liking of the Nexus (though it doesn't seem unreasonable to me, since the same amount of data is being downloaded and the demand spread across the available servers - it only makes one connection to each server.)  I'll have to speak to the admins before unleashing a possible Nexus-taunting device on the STEP forum.

So long as Dark0ne & team don't have an issue with this. It would be a problem from their viewpoint if it circumvented too much of their framework as to cause inaccuracies in the statistics that they gather. As long as each user's ID is attributed to all activity, it would seem OK.

#8 d2allgr

d2allgr

    Prisoner

  • Members
  • 9 posts

Posted 25 January 2013 - 04:25 PM

well, so far from my limited messing about with the nmm code (limited because I am just know getting to know c#, I am mostly an sql kinda guy) I managed to interface with the mod/file info side of things and managed to test out the multipart downloader that is included with the nmm and rip it out, using it in a purely console app that downloads a list of files from an xml spec (specifying a list of mod ids and the corresponding file ids to download). It seems quite easy to change the downloader as well to use a list of urls instead of single one (I found a function that returns all the download locations instead of just the closest one, similar to the website). So together with the cookie that the login interface generates, u could probably feed both the aria2c or any other similar download utility and work from there (assuming DarkOne doesn't take any offence to us stress testing his servers every time we decide to download the list of mods in STEP :D )
  • 0

#9 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 25 January 2013 - 06:09 PM

@d2allgr Sounds like you've found some very useful stuff. My script just uses wget to login and retrieve the cookie, then Aria2 picks up the cookie and does the downloading with a lot more style than poor old wget. The script just appends a list of the mods file IDs to a list of known server paths, and generates a download list (with mirrors for each server for each file) and passes it to Aria2 to do its kung fu. One problem I foresee is that it overcomes any drawbacks of using the free servers, as opposed to the premium ones, including multipart downloading and resuming. During the download of the entire STEP collection, many errors were generated and files disconnected half finished, but it just makes new connections and keeps trying until the job is done, at good speeds. EDIT: All that stuff that you've found could be very useful to the Semi-Automatic Thunderbolt Thingie. Ultimately, I would think what's needed is a STEP friendly manager and downloader that can deal with the Nexus in an acceptable way. We could perhaps just strip down the aria approach so that it makes only one connection to one server, but at least will keep retrying and reconnecting a large download list in a reliable way.
  • 0

#10 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 30 January 2013 - 09:33 PM

I discussed automated downloading with Dark0ne, and it's not something he would like us to do for the foreseeable future. I feared as much, and it would indeed screw up the way the Nexus registers downloads, endorsements, etc. He suggested we could do something similar with a plugin for NMM, but that this might mean reduced ad revenue, with fewer eyes on the site. I told him we wouldn't want to do anything that harms the Nexus, so I think automatic downloading should be considered a non-starter. I did, however, suggest that he might like to implement mod collections formally on the Nexus, with perhaps automatic downloading of the collections as a premium feature. I'll let you know what he thinks of that.
  • 0

#11 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,258 posts

Posted 30 January 2013 - 10:13 PM

I discussed automated downloading with Dark0ne, and it's not something he would like us to do for the foreseeable future. I feared as much, and it would indeed screw up the way the Nexus registers downloads, endorsements, etc. He suggested we could do something similar with a plugin for NMM, but that this might mean reduced ad revenue, with fewer eyes on the site. I told him we wouldn't want to do anything that harms the Nexus, so I think automatic downloading should be considered a non-starter.

I did, however, suggest that he might like to implement mod collections formally on the Nexus, with perhaps automatic downloading of the collections as a premium feature. I'll let you know what he thinks of that.

Thanks for reaching out to him Monty ;)

I suspected as much as well, so it is good to have verification that this would be overstepping ... before overstepping. It a way, it is just that we should need to work through the Nexus, as it is like a right of passage that people should need to download each mod individually and contemplate the $65 donation for the lifetime premium membership.

Well worth it, I can say.

#12 d2allgr

d2allgr

    Prisoner

  • Members
  • 9 posts

Posted 31 January 2013 - 03:03 PM

Well, was worth it as an experiment and a way to learn some c# in the process If DarkOne ever changes his mind, the tool at the moment can (coded and semi tested): - Download all the files of a single mod. - Download a specific mod file. - Download a collection of files across multiple mods, based on a xml input file. - Use up to 5 connections to a a single regular server or up to 5 connections to any 5 random premium servers depending of the user's status. - Retry failed downloads (no resume yet). Until then, I will keep it private (and not use it ofc) to avoid a ban.
  • 0

#13 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,258 posts

Posted 31 January 2013 - 03:37 PM

Nice work anyway guys. We could certainly use help programming the simplified GUI for managing STEP (or any modded setup). Start date TBD

#14 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 31 January 2013 - 03:39 PM

Sounds brilliant. It is a shame, but I can understand it. I definitely think the Nexus is missing a trick if it doesn't come up with an approved method for dealing with mod collections, especially if auto downloading was an incentive for premium membership. Hopefully we haven't heard the last of the story. I'm a bit busy wrestling with the Creation Kit in my spare time at the moment, but I'd still like to look at a system for automatic STEP installation, defined by simple instructions in XML files that would describe the extraction and combination of mods. If you feel like stretching your c# a bit further, please do look into that side of things. I find that sometimes a bit of duplicated effort is not such a bad thing, especially when coordinating a project like this - sometimes the difficulty of synchronizing work can be a block. I think, in some cases, each doing their own thing, then bringing it all to the table and merging the best bits works well. Rather like how STEP and Neovalen are working asynchronously in parallel. EDIT: This was also posted asynchronously in parallel with Z's post!
  • 0

#15 stoppingby4now

stoppingby4now

    Sleepy

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,281 posts

Posted 31 January 2013 - 11:17 PM

Sounds brilliant. It is a shame, but I can understand it.

I definitely think the Nexus is missing a trick if it doesn't come up with an approved method for dealing with mod collections, especially if auto downloading was an incentive for premium membership. Hopefully we haven't heard the last of the story.

I'm a bit busy wrestling with the Creation Kit in my spare time at the moment, but I'd still like to look at a system for automatic STEP installation, defined by simple instructions in XML files that would describe the extraction and combination of mods. If you feel like stretching your c# a bit further, please do look into that side of things. I find that sometimes a bit of duplicated effort is not such a bad thing, especially when coordinating a project like this - sometimes the difficulty of synchronizing work can be a block. I think, in some cases, each doing their own thing, then bringing it all to the table and merging the best bits works well. Rather like how STEP and Neovalen are working asynchronously in parallel.

EDIT: This was also posted asynchronously in parallel with Z's post!

Take a look at AHC, as it was doing exactly that (and was working at one point). It's written in Visual Basic though, and the last amount of code had some bugs. The interface could also use some work, but it was definitely headed in the right direction.


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users