Jump to content


Photo

Automated STEP Installer - Now in Beta


  • Please log in to reply
191 replies to this topic

#16 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 11,933 posts

Posted 15 October 2013 - 03:53 PM

Sounds great but not sure how this would work with MO. I have a STEP:Core profile, a STEP:Extended profile and then my personal profile. The Core and Extended profile only has mods active that are relevant to those profiles; however, my personal profile has mods that are not part of STEP. The issue I see popping up is the MO keeps all of these mod in one location. If this tool is only looking at these folders then how would it work with all of these extra mod folders tossed in. Not to mention that the folder names might not be the same as the archive name since MO makes it super easy to rename the mods at installation. I suppose one way around this would be to maintain separate archives outside of MO.

#17 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 18 October 2013 - 05:39 PM

I have though quite a bit about the mod maintenance problem, and I think the best solution is not to try and maintain mods in a real time bases. The automated STEP instructions would be mod version specific. Most mod authors do keep old versions of the mod on the download page so this is maintainable . For the majority of STEP mods, which are texture replacers, it matters very little what is changed between updates. The worst case is STEP misses out on the newest additions. For scripting mods it can be problematic, but nearly all of the most popular script based mods have well archived download sections. Whether or not a script is included in a STEP installation could perhaps be contingent on how consistent the author is about archiving. On the implementation side, I think it is possible to create an automated installer that is not strictly version specific. If the script or whatever use wild card in the archive description, it need not be concerned with the exact names or specific file structures in an archive. Most of the time when a mod is updated the internal file structure remains the same. So long as the STEP user places archives in the correct location, the script need not be concerned with what the mod is called, just that it unpacks whatever is there and moves whatever files that it is instructed too. I think what is realistic is that whatever automation comes to be used will be updated as the STEP guide is updated. A regular release schedule of say, once a month or every two months would be much more manageable for a small group of people. I would also like to point out that the skyrim modding community is well aged at this point. The peak happened a long time ago, and that is ok. These problems with maintaining and updating mod lists is very common to many modable games. If we can develop a powerful and flexible system it can benefit lots of people now and in the future. Imagine if ES VI comes out and we have a working and full featured automated installer day 1. Monty, I want to help with your project. I don't have a lot of time to put into development right now. I am taking a lot of units and part time job and blah, blah, blah, but some time after the holidays I would love to get my hands dirty with something. For me there would be a certain sense of satisfaction in just working on the project for the projects sake. Feel free to burden me.
  • 0

#18 TechAngel85

TechAngel85

    Akatosh

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 11,933 posts

Posted 18 October 2013 - 08:55 PM

I would disagree that it's peaked. There are a lot of great overhauls still coming out and even more being refined and finalized. Even the Unofficial Patches are churning out updates regularly (not so much recently because of their move to their new format). I think it'll be at least another year before we see Skyrim modding peak and start to slide off.

#19 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,249 posts

Posted 19 October 2013 - 01:28 AM

Agree. The modding scene may have peaked in terms of the number of mod authors contributing, but the evolution of Skyrim mods still has not reached is crescendo. Lots of complex overhauls are still being actively developed and more brand new ones are sure to come.

@mothergoose
I think it would be great if you take up work on Monty's project. We need all the help we can get, and any experimental work is sure to bring gains in some way, even if they are tangential. I think that this should be picked up where Monty left off, and don't forget that Wrye Bash and MO both have classes and functions that are able to look inside archives to retrieve relative file paths and build lists. If you know Python and C++, you would be able to port those chunks into your application. The active mods in any given version of STEP:Core can be retrieved using this Semantic query (which could be further developed if we ever consider tracking mod versions).

Semantic Mediawiki could likely be used to spit out data to a program, but I am not knowledgeable enough to say for certain. s4n knows how to script and also about the SMW API.

If you are interested in actively pursuing this development, we can add you to our staff in the appropriate role. Then you could recruit persons to help out, but that is very difficult ... lots of eager people with low level of follow-through and commitment. We have been disappointed many times with aspirations of help only to watch initiatives either never get started or get abandoned due to lack of leadership or support.

The few of us that handle most of the work around here are pretty much consumed with maintenance of the current guide and support infrastructure.

#20 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 19 October 2013 - 03:27 AM

Well I have a lot of ideas about how to implement my version of an automated installer. I have to admit Monty's program sounds really powerful, but I don't know very much about it. Sending him a PM now so we can hopefully talk more about it. I can say with some confidence that if I have any tool to print out the files in an archives I can write a C++ program to do the rest in terms of generating a file list. I have some experience with scripting in unix, so although I don't know how to do scripting in windows, I could probably pick it up pretty quick. Programming doesn't change much, just the syntax. I won't know until I am knee deep in the code, but conceptually I have a plan for how to tackle this problem. 
  • 0

#21 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 19 October 2013 - 09:08 AM

I'll try to help you harness my stuff if you'd like, but of course you quite welcome to try a wholly different approach. You mention a non version-specific way of doing it, and that it certainly possible, if you start from scratch. I did not feel that this was a good approach, for the reason that automation is fundamentally version-specific. That is to say, you could only be confident that your system will work correctly for the specific versions for which you create the patch; it is certainly possible to create a mechanism that is version agnostic, but it may then tolerate unwanted outcomes if it applies the original logic to a new version that may be significantly different. In order to be confident of the system, one would still need to check the compatibility of each update - an examination that might well take longer than the generation of a new patch, in my scheme. It seemed to me that being specific about exact versions, for which the patches are precisely tailored, would have the added benefit of certain, error-checkable output. By creating a version-tolerant system, the only real advantage would be that patches may be re-useable - but time would still have to spent confirming this for each update, and you no longer have the benefit of a consistent output.
  • 0

#22 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 19 October 2013 - 12:31 PM

Well the automation would have to be version specific. I think a non-version specific implementation makes sense because it gives the STEP user some tolerance. It also makes the maintenance side a tad easier, because then the script doesn't have to know specifically which rar files to unarchive. More than one archive could be placed in a mod folder and they can all be extracted, making the installation process a bit more natural also. The way I had it in mind is that the STEP automation instructions would be version specific, and if the STEP user followed them correctly there would be guarantees about the exact output because the contents of each archive would exist somewhere in a file list. If the STEP user didn't follow the instructions and instead downloaded a different version, for most mods in STEP, this would be rather harmless. If HD2K adds a dozen more textures in a new update from a previous version of the automated installer, than if a STEP user places that more recent version in a mod folder instead of the specified version than the script would probably still work as intended. That is the the relative file path for all the other textures that existed in the previous version would very probably still be intact.
  • 0

#23 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 19 October 2013 - 02:58 PM

You would get away with it for the majority of cases, but I think you'd find enough cases where it becomes problematic. (One example of the top of my head - WATER 1.5 to 1.6 was a complete rewrite of the whole mod. This sort of thing, and significant restructures occur often enough to be problematic.) The other thing that you lose without version specificity is the ability to apply binary differential patches - to automatically patch in cleaning edits to ESPs, and so on. If you decide to take a different route, that's cool, but I would suggest you look at having some sort of engine to automate the generation of scripts/patches/whatever. If you go down the road of hand writing a line-by-line, file-by-file script for each mod, it's going to be a VERY time consuming task.
  • 0

#24 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 19 October 2013 - 03:54 PM

My plan is to use 7z to output the contents of an archive to a text file, and then write a C program to generate a master file list from these text documents. The C program will also be able to compare different file lists to each other and the masters list. A script or c program will read the master list to generate a modded skyrim file structure and archive it.
  • 0

#25 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,249 posts

Posted 19 October 2013 - 04:21 PM

Agree that versions will throw a wrench in things at least once a week (and possibly much, much more). C++ and Python can handle complex logic with complexity decreasing rapidly with consistency of input. Why not account for version (or even better, file modification date) in automation via the Nexus API, then allow this to trigger a search of the archive and compare relative directory strings in order to determine if the file should be interpreted as an identical CRC. Then the report could spit out a list of mods and associated changes. At the very least, we have marching orders for changes, but why stop there? Use Monty's application (or analogous) to do that work as well. That is how I see the STEP:Core installer as an example (all files triggered to download via wget, of course).

#26 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 19 October 2013 - 05:11 PM

I don't think its a good idea to implement any automation in the downloading. The first reason is the nexus would be out a lot of add revenue because no one is actually visiting the site and they would still be fronting all the cost of bandwidth and file hosting. The second reason is that unless a person actually visits a mod authors page they can't endorse, donate, comment, or provide any of the community spoils to the mod authors, which is so important for the community. If you meant an automated download for the guy maintaining automated STEP I think that is a cool feature. I will readily admit though I don't know how to do it. MO and NMM mod manager do keep track of mod updates already so I also questions how necessary it is to re-implement that functionality.
  • 0

#27 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,249 posts

Posted 19 October 2013 - 06:42 PM

I don't think its a good idea to implement any automation in the downloading. The first reason is the nexus would be out a lot of add revenue because no one is actually visiting the site and they would still be fronting all the cost of bandwidth and file hosting. The second reason is that unless a person actually visits a mod authors page they can't endorse' date=' donate, comment, or provide any of the community spoils to the mod authors, which is so important for the community. If you meant an automated download for the guy maintaining automated STEP I think that is a cool feature. I will readily admit though I don't know how to do it. MO and NMM mod manager do keep track of mod updates already so I also questions how necessary it is to re-implement that functionality.

Yes, I mean automation for we who are administrating this, it would be a huge lifesaver for keeping up on maintenance. I think that this is worth building if it will be used, and I think a prerequisite is maintenance cost. Keeping it as low as possibile is a huge boon to all in the end.

Reimplementing within the app makes sense, otherwise, there is a need to depend on another app or to get the data from that app. This seems like more work. Why not port the routine from MO, which is coded in Python and C++ already I think? Then the app gets the data without needing another app (or person). The admins here always wind up visiting the STEP mod Nexus pages a lot in order to update notes, etc. This helps a bit.

#28 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 20 October 2013 - 03:08 AM

 Why not account for version (or even better, file modification date) in automation via the Nexus API, then allow this to trigger a search of the archive and compare relative directory strings in order to determine if the file should be interpreted as an identical CRC. Then the report could spit out a list of mods and associated changes. At the very least, we have marching orders for changes, but why stop there? Use Monty's application (or analogous) to do that work as well.

That is very interesting, and this sort of update monitoring would fit very nicely with the "mod library organiser" concept I was talking about. 

Hmm... it seems there's life in this scheme after all.  I may have to return to my lair and see what can be done.
  • 0

#29 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 20 October 2013 - 12:56 PM

I feel really dumb for not having thought of this before... MO and NMM keep track of exactly every file in a users skyrim data folder. It also keeps track of the archive that it came from, for every single archive. For NMM, looking at this organized list is a simple as clicking on the folder icon, and selecting "Open NMM install info folder". It keeps all the info in an installlog.xml document. A C program could be written to parse the installog in NMM or MO and recreate its file structure, thus recreating the exact modded skyrim data folder file structure. On the maintenance side, all a STEP maintainer would have to do is run the c program on their installation log, and use the mod organizer to create the version of automated step, same as they would have done before. If STEP wanted to account for user without a DLC or some similar configuration, no problem. Create that installation locally with NMM or similar, run c program on the installlog. That easy. Mods have been updated between releases.... update mods with NMM or MO, run c program again, done. Here is my NMM installlog. It is organized such that the archive is read in and the mod name is recorded. Th archive is given a string, something like "0adve". This string refers to the files in that particular archive, complete with the relative path.  http://www.mediafire...tbh1bzs5tdxrg2b
  • 0

#30 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 20 October 2013 - 01:17 PM

That's really handy if you're keen on creating a brand new solution.  But just to clarify, in case you want to avoid duplicated effort - the patches created by Skyrep contain the same information, in the form of a portable BerkleyDB.
  • 0


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users