Just a quick update, made an installer. Can be found at https://yamm.thelazy.net/mods/
Status of this?
Posted 11 June 2015 - 12:17 PM
OK, I have a couple of questions about setting this up for use with MO (or stand-alone).
- I have installed and added the three available services you provided. Where are the JSON files located in my filesystem? I cannot find them under the YAMM/MO paths, nor do they appear in %USERPROFILE%/YAMM.
- I see the YAMM client in the MO plugins menu, and the service is started. How do I use this under MO? Should I add YAMM EXE as a program to launch from within MO?
- Where is the plugin file(s) for MO located? EDIT: OK, I have determined that the plugin file is <MO>/plugins/plugin_MO.py and this appears to be sourced by Tannin for plugin devs, and Terrasque modified for YAMM. Any way to rename to YAMM.py or something (eventually)?
- What config file controls the plugin attributes (version, description, etc) as seen under MO (Settings > Plugins tab): The Description says: "Installs mods and doesn't afraid of anything" (want to see if I can change "doesn't" to "isn't" ... just for kicks [you may want to edit in your next update])
- I created a new "YAMM Testing" MO profile for testing mod download and installation. Not sure if it matters. EDIT: I found the string to change within the plugin file:
def description(self): return "Installs mods and doesn't afraid of anything"
- Are the instructions outdated now? This version seems to use EXE and not PY for app launch.
Once I understand the basics of using this, I will test with my own STEP JSON.
Posted 11 June 2015 - 01:12 PM
1. The data from the json is stored in an sqlite database, the data/modinfo.db file. storage.py holds the structure for it, but you can also use an sqlite3 browser to browse the data in it.
2. As long as MO is running and the plugin is loaded, YAMM will be able to talk with MO. It does this via an mmapped file. And yeah, that one should be renamed :p
3. And the plugin exports some values when it's loaded. Most of that is just dummy data I put in when trying to get the plugin to work, so should probably be fixed :D
4. It doesn't matter that much, but it makes it cleaner. I still have a lot to do on the MO plugin, but I really want torrent support and solid up some of the yamm core systems first (including adding a settings system)
5. And the instructions are .. well, not outdated, but the installer does all that. If you download the source you need those steps to make everything work.
Posted 11 June 2015 - 01:47 PM
Thanks for the response ...
OK, I have figured out that YAMM must be started independently and that the MO plugin only allows YAMM to talk to MO. To me, this means that the YAMM EXE can be registered to execute via MO ... sound reasonable? Alternatively, you could eventually implement something to establish YAMM as a preconfigured executable for launch from MO.
I wanted to test a mod with several dependencies, so from YAMM, I chose ArousedCreatures ...
... which has several interdependencies. The mod attributes pops up (from JSON, I assume) in a new YAMM window with option at bottom to "Download mods" ...
I click that, and get a new window with my "Mod Download List" (also presumably from JSON defs), including ArousedCreatures and all dependencies checked ...
If I click on "Send checked mods to Mod Organizer", MO gives me an error (repeatedly for each checked mod, so had to say "OK" 6x) ...
Alternatively, if I select "Download checked mods" from this window, mods are downloaded without issue into %USERPROFILE%/YAMM/files. THEN if I click on "Send checked mods to Mod Organizer", I still get the error, but after clicking "OK" in the error window, the MO installer is invoked. This behavior is reiterated for each mod in the list for which I have a file downloaded (again, this iterates 6x).
- I get to user Nexus premium servers when downloading using Nexus downloader (either directly or from within MO) ... does YAMM proxy my preferred servers from Nexus or my MO config? If not, that would be a basic enhancement I think.
- Ideally, I would rather not download mods into %USERPROFILE%/YAMM/files, but rather into the download folder I have configured in MO under Settings > General > Advanced ... again, this is just something to conside3r once you are out of the 'alpha' dev stage of this app.
The basic concept works though, so I like that we can configure and enable multiple downloads according to our JSON preset. Would just like to figure out how to alleviat the MO error. After this is fixed, I will create a JSON for STEP and test for a large list (having use of premium Nexus servers will be a huge benefit then).
Posted 11 June 2015 - 02:00 PM
First of all, it's not using the nexus servers at all. I'm currently hosting all referred mods on my own server. Would be nice if I could eventually persuade Nexus to support it, but I don't have much hope for that at the moment.
The MO error seems a bit weird, functionally speaking it should be exactly the same as opening the archive from MO, except the default name suggestion.
Posted 11 June 2015 - 02:18 PM
OK, I forgot about the mention that this does not use the provider's download mechanisms (duh, that has been our main issue, and I always am mentioning it ... guess I am just spacing on my own self >< ).
It is still an issue we need to figure out, but even without the auto-download feature, just providing a list with the links in a particular order will be a good thing for us aside from file hosting issues. Torrents are nice for many things, but it would be a breach of our protocol around here to circumvent the host's tracking/advertizing mechanisms (given we represent a community).
Posted 11 June 2015 - 03:28 PM
but it would be a breach of our protocol around here to circumvent the host's tracking/advertizing mechanisms (given we represent a community).
Yeah, that's a headache... On one hand I don't want to piss off anyone or have people think I'm stealing or anything.. But on the other hand I kinda need something to work with and to demonstrate with. So I've grabbed a small sample of related mods for that. I haven't checked all of them, but for those I looked at there was nothing in description for redistribution, or specifically saying that there was no limit or that if you do then credit mod maker.
SKSE have a very strict no-distribute policy, so that one has an empty entry. That also let me test how the software worked with empty entry :)
As I have mentioned, it's technically possible to add nexus id for mods and let MO download them from that instead of hosting, but firstly that would increase the load of the Nexus servers and taking bandwidth without giving anything back, and secondly that would tie it directly with MO and make it less useful to run alone or with other managers later.
Posted 11 June 2015 - 06:46 PM
I do see the general benefit though and applicability to many various content providers, but it would seem like you either need a vast 'underground' hosting service or a collaborative business agreement with a legit host.
I suppose I like the idea of automating mod downloads, but we just can't do it that way. From your position as the app dev, and from STEP's position as a large conglomerated user of your app, we cannot (without pissing Nexus off) enable users to avoid Nexus page views by circumventing their site altogether. Single users will love it and there is not much Nexus can do unless they encrypt or periodically change their file links or require some valid user token prior to accepting a file download transaction.
Our best bet is to create a plugin specifically for use with MO, which already builds in a useful and 'polite' mechanism for mediating file downloads by requiring users to first visit the file source and acquire the mod ... MO can automatically associate mods and mod meta information (by clicking on Nexus "Download with manager" link or allows the user to do it semi-automatically or manually). Once the mod is in MO, then the user can easily visit the source site or conveniently even update/endorse the mod (which does circumvent the Nexus, but only after an actual visit).
What is missing from MO (and all mod managers) is pre-populating MO with the desired mod list and dependencies (and possibly prioritization order, etc.). You have the beginnings of a tool to do this, but it probably requires more elaborate MO plugin development. Other mod managers suffer the exact same restrictions, but they are probably not nearly as simple or accommodating for you to build against or extend.
Outside of community modding, I could see the usefulness of your tool, but only for mod authors or those legally hosting their own or publicly distributable content that have interdependencies and very stable links (like Linux and CRAN mirrors).
What is your main objective with this program? Can you describe one or two particular use cases?
Posted 11 June 2015 - 07:05 PM
This is how I picture semi-automatic STEP:
- A plugin in MO probably, or perhaps a helper app with MO plugin
- Starts off with a dropdown of downloaded Guides
- Once the Guide is selected, the plugin scans MO's download directory for the correct mod files specified in the Guide (determined by filename\filesize\CRC?)
- All files found are marked Ready! and all files not found Download or Select File
- Clicking the download button takes you to the mod page and instructs the user which file to download (perhaps using the built-in browser in MO?) which automatically is taken over and downloaded into Mod Organizer and subsequently marked Ready!
- Once all mods have been collected, the plugin shows a Install Guide! button that automagically installs all the mods using the options specified in the Guide.
Posted 12 June 2015 - 05:21 AM
From your position as the app dev, and from STEP's position as a large conglomerated user of your app,
we cannot (without pissing Nexus off) enable users to avoid Nexus page views by circumventing their site
Correct, which is why I haven't added some features that would make this easy.
Single users will love it and there is not much Nexus can do unless they encrypt or periodically
change their file links or require some valid user token prior to accepting a file download transaction.
Actually, there's not much nexus can do. Unless they add captchas, then as long as a user can download it, then a machine can too.
And captchas are both a pain for everyone, and becoming less and less effective.
What is your main objective with this program? Can you describe one or two particular use cases?
Well, let's start a bit with myself. I'm a web / server coder, both by profession and hobby, and I've been doing that for almost 15 years now.
My current job involves things like scraping webpages, server-to-server communication, API delivery and consumption, scaleable distributed computing..
Mostly collect, reformat, organize, compute on, present and send data in various forms. And 90% of the work is done on linux boxes, mostly debian.
And that's the "frame" I'm used to work with and think about problems in relation to.
Now to this tool. The idea started when I was curious about an LL mod a friend linked me. So I downloaded it.
And the requirements.. And the requirements' requirements... And... Eventually I had downloaded 14 mods. Just to get that one working, and that was only the required ones.
It also turns out, I missed one requirement, which didn't crash the mod, just made it behave oddly. And that was the point where I started thinking "It has to be a better way than this.."
I started working with some ideas in my head, then found this thread.
Reading through that and seeing so many wanting something like what I was thinking made me consider it more seriously, but I didn't want something LL specific.
I also wanted something that was both easy to implement server side and really light on resources, with good options to scale
(the filehash, download urls list and torrent parts are a direct result from that). So I started testing out my ideas and built a simple CLI program which eventually evolved to what I got now.
So my main objective is to have a simple way to one click install a mod and it's dependencies from a link, without it being tied to any specific site or page in any way,
that's easy to add support for on a site, and be able to scale up cheaply.
Regarding performance.. The metadata, the meat of the system so to speak, is just one static file. Modern hardware and http servers can serve tens of thousands requests per second more or less out of the box.
The file serving can be done via torrents or via voulnteer hosts, and could easily even be (semi-)automated at that. The hash will ensure that no tampering of the file have happened.
And searching and dependency resolving and calculations are done on client side, thus not impacting the server.
Posted 12 June 2015 - 12:54 PM
I looked at the LL thread a bit, and it seems that your solution would be perfect for that kind of project.
- Potential LL users navigate to an LL affiliate that provides a link to the LL metadata - as you said, hosting the metadata is super easy and requires very little bandwidth, so it has essentially no cost other than maintaining/syncing the JSON(s) as mods are modified/removed/added (which can be a lot of work, depending)
- LL JSON is used by your program to download mods (and dependencies) the user is interested in - Unless all mod contributors agree to provide their mods through some personal/community file hosting service or via torrent, there is no way to acquire the files without potentially pissing someone off (and potentially getting into legal trouble). The LL community may be small and 'tight' enough to make it work.
- Install the mods according to other metadata descriptors - Hence, this program would either need to be a mod manager itself or a plugin to an existing mod manager.
The trouble is #2, and it cannot be avoided unless you have an agreement with the file hosting service and/or all contributing mod authors. Assuming that no significant, reliable, comprehensive, free file host like Nexus would ever agree to allowing a third-party app to largely obviate user visits ... and assuming that getting all/most mod authors to approve distribution outside of their own chosen host or via torrent is impossible ... #2 is the deal breaker for automated mod download, and it just can't work.
What can work is just what DoubleYou and I have posted:
- Maintain a database of mod interdependencies and metainformation (hosted by almost anyone, anywhere, any way)
- Develop an MO plugin to utilize this information to facilitate manual download through MO and automate installation/configuration in MO
The concept of "Automated STEP" was changed to "Semi-Automated STEP" early on when we realized that automating mod download created by various unaffiliated authors without consensus agreements in place is just not possible. We can theoretically automate the installation and configuration but not the acquisition.
Your solution would still be invaluable though for the reasons you state above: Too many ill-defined requirements and configuration exceptions based on mod interdependencies ... "there has to be a better way". Isn't the biggest problem understanding how to put it all together correctly? ... rather than acquiring the mods? That's precisely why we created STEP: to simplify modding to allow everyone to achieve a 'good' result without the headache of the time requirements of trial and error and uncertainty.
All we need now is to integrate elements of our guide into useable meta-information that allows the mod manager to assemble everything that we now instruct our users to do manually using the mod manager. This would save many people hours/days/weeks of time reading/reinterpreting/misinterpreting instructions, asking questions, and supporting the less competent of the user population.
Just think how such an application could be helpful for various mod authors ... they compile their own JSON (or table that can easily be converted to JSON) with config details/exceptions for their mods as well as the interdependencies with other mods (rather than explain it all haphazardly and get lots of questions to answer and all the angst that comes with user confusion).
Posted 13 June 2015 - 06:09 PM
Assuming that no significant, reliable, comprehensive, free file host like Nexus would ever agree to allowing a third-party app to largely obviate user visits ... and assuming that getting all/most mod authors to approve distribution outside of their own chosen host or via torrent is impossible ...
That's two pretty big assumptions, tho. In the same post, you already pointed out two good arguments for that happening. And the "endgame" for this idea is exactly that.
As for the two good reasons for:
1: "hosting the metadata is super easy and requires very little bandwidth, so it has essentially no cost other than maintaining/syncing the JSON(s) as mods are modified/removed/added" - and the syncing would happen automatically if the site supported it. Server cost is a big problem for large sites, and this could potentially make a huge impact there. Authentication and download tracking is not impossible with this system, but it'll potentially complicate things (which I have avoided for now)
2: "Just think how such an application could be helpful for various mod authors [...] with config details/exceptions for their mods as well as the interdependencies with other mods (rather than explain it all haphazardly and get lots of questions to answer and all the angst that comes with user confusion)."
And keep in mind that a service providing mods through a low-resource system and relying on torrents could potentially work out from a sub-$50 per month server. And likely on much cheaper servers too, the main cost would be storage. And since different services largely works nicely with each other it could be spread out over the community and still represent a "complete" system from a user and technical point of view.
Basically, there's no technical reason why it won't happen, it's only politics and inertia.
When that's said, I have been thinking about replacing the download checkbox with a button opening the homepage of the mod. If that link pointed to the Nexus page for that mod, and with MO's handling of Nexus links, it would be close to semi-automatic.
3 - Install the mods according to other metadata descriptors - Hence, this program would either need to be a mod manager itself or a plugin to an existing mod manager.
Develop an MO plugin to utilize this information to facilitate manual download through MO and automate installation/configuration in MO
Isn't the biggest problem understanding how to put it all together correctly? ... rather than acquiring the mods?
NB: I'm a newbie to mod installation, and this is mostly what I've gathered from googling and looking at mods and tutorials
This is a complicated problem, but also a problem that has been worked on and tried solved for a long time. We have for example FOMOD format that can give some tweaked setup,
and things like LOOT and TES5Edit that can to some extent minimize or resolve conflicts between mods.
So there's already a lot out there for these kind of things, and it's not an easy problem to solve. Compare that to what my project is trying to solve, which from what I've seen haven't had any decent attempts to be solved before, and thus easy to make big improvements on.
In other words, this part might be the biggest challenge. I can see some potential solutions, like providing an external FOMOD file for a mod when installing or have a mod late in the load order that only changes the settings of other mods or overwrites certain files (relatively painless in MO's virtual file system).
As for hosting mods, I could set up a mod hosting service on my server, and could maybe set apart a few hundred GB for mods storage. You'd still need to give mod authors and users a reason to use it, and maybe convince some people that they should at least allow yamm: type links in their forum / comments systems.
Easier installation, fewer non-important questions for mod authors (like people forgetting a dependency), potentially better and more stable download speeds with torrents .. that could be some valid arguments.
A small brainstorm. You could compile a list of mods used in STEP and send the mod authors a message with something like "hey, we recommend your mod in our STEP mod guide, but the technical parts is pretty challenging for some of our users. _To make things easier we are trying to automate what we can of it, and we wonder if you would allow us to redistribute your mod for that purpose?" and record yay/nay/no answer. If you could get .. say, half as "yay" that could help a lot for the process, and maybe it would be possible to create a fully automated "lite" version too.
Edited by Terrasque, 13 June 2015 - 06:10 PM.
Posted 13 June 2015 - 06:29 PM
The big thing with that is with all the absent mod authors
Posted 13 June 2015 - 07:48 PM
Actually, the two assumptions I made are pretty solid assumptions:
- A comprehensive/dependable host (Nexus) will not like us directing users via this tool (let's face it, ONLY Nexus is really relevant, they have 90% of all mods for TES games ... nobody else comes close besides Steam, and they lack a ton of mods too).
- There is no way that even half of mod authors will agree to allow their mods to be hosted outside of Nexus. Even if they did, there would be the major maintenance problem of syncing mods on the alternate host as Nexus mods are updated. This would be impractical, and syncing with Nexus is same issue as #1. Lastly, as DY states, there are many MAs missing in action.
If 100% of relevant mods to STEP (let alone Nexus mods) are not hosted (and updated) on your server, then the convenience of the automated download is a total wash in my view. It's either all or nothing, because exceptions require special treatment, and the goal is to keep it simple. MAs will also object to their mods being supplied via torrent, and the same issue of syncing updates still apples.
The real issue that your program has the potential to resolve, as I said, is the problem of prepopulating the mod manager with the relevant list of mods and instructions for installation order and methods (some are FOMOD, some are manual, some have other types of installation, some require package reorganization, etc.) ... hypothetically:
- Your plugin adds a new tab to MO for mods not yet acquired, including metadata supplied by the JSON. MO treats these as "potentially installed" mods, including some of the functions available for installed mods in the standard left pane via the context menu.
- The user uses the context menu to review metadata (e.g., visit Nexus mod page, dependencies, mod categories, file structure, etc.) and navigates to Nexus to download each mod.
- Click on [ Install Mods ] button supplied by your plugin, and all these mods are installed, disappearing from your custom MO tab and reappearing under MO's left pane (since they are now just installed mods). Instructions in the metadata supply installation/repackaging for MO.
- Everyone profits. Users trust that all is installed correctly, and our support staff can trust this as well and focus on real support issues.
I just don't see any scenario where "automated download" of mods can ever work for anything but small, private group projects hosted by servers administrated by the same group. In fact, I absolutely know that it is not practically possible for all of STEP mods much less all of Nexus.
... your tool is still very interesting though for these other reasons, but this functionality is quite an extension of your original idea. If I were a programmer, I would definitely build the plugin to which I allude, but I can only hope that someone else (e.g., you ;) ) has the interest in doing so. I have some programming experience, and I can assist with testing and grunt work, too.
Posted 16 June 2015 - 06:39 AM
We'll see how it goes. Since automated downloads are the main goal of this project, I'm going to focus on that.
When that's said.. Easier installation of mods is a sub goal, but lower priority.
Part of that is because various other programs already do a lot of that work, and it would probably mean either changing them substantially or
reproducing a lot of that work just to get the base functionality needed to start trying to handle those problems.
In addition, mod install nuances is also a field where I don't have that much experience with yet.
So we'll see how it goes :)
Edited by Terrasque, 16 June 2015 - 06:40 AM.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users