Jump to content


Photo

Automated STEP Installer - Now in Beta


  • Please log in to reply
191 replies to this topic

#136 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,313 posts

Posted 20 November 2013 - 01:35 PM

Regarding the current ASI process and config maintenance: I think that this will get the STEP-team attention it deserves once we have a maintenance procedure figured out. To me, it seems like Windows shell scripting is not necessary if this is configured as a MO plugin. Then, MO could be used to reconfigure the file structure (or remap it, or whatever is happening in MO when one has to use the MO GUI to indicate the directory structure interpretation). MG, you should install MO and work with it a bit to see how package reconfiguration works within MO. I think that this is the only realistic way that we will be able to manage mod reassembly instructions. Workflow needs to be consolidated into a single application. I am happy to continue testing POC of each release, but until it is integrated with MO (I just started using MO and experienced once such case where I used MO GUI to reconfigure/remap an incompatible file hierarchy), I can't realistically commit the STEP team to owning, even if we have a volunteer. The reason is that if any such volunteer stops maintaining, then I or some other dedicated site admin will need to step up to fill in the gap. I think that Pack authors will also more readily take advantage of ASI once it is integrated into MO using MO directory remapping/reconfig methodology. So far, the general methodology has promise, but I am hoping for a mechanistic alteration that leverages MO facilities or something directly compatible with MO (i.e., XML-based instructions IIRC). I see a lot of potential for this as a general modding tool for STEP, Packs and any mod-installation process in any MO-compatible game, so I hope that developers like you continue to move it in this direction.

#137 wolverine2710

wolverine2710

    Mod Author

  • Contributors
  • PipPipPipPipPipPip
  • 441 posts

Posted 20 November 2013 - 01:47 PM

@mothergoose729. I will give a far more detailed explanation later (tomorrow morning) but a smaller one (yeah right) now. I think you have done a great job, learned a lot, learned what works and what not and hopefully have enjoyed the coding part. You seem to have hit the wall called maintenance.  This has been mentioned a few times.

Perhaps its time for a new start with ASI and leverage the tools which exist. What has been discussed in the last week is that maintenance can be minimized by letting do MO the hard stuff. As in a packager creates a certain setup in MO which consists of mods with and without installers, with and without difficult to parse  badly packaged mods. After that he selects the MO plugin 'export ASI config' which creates an XML file. This XML contains info like nexusid, version, category, modname (in MO), filename of the zipfile AND for each mod all files associated with it. When a users want to install a complete S.T.E.P. 2.2.7 he starts MO and selects the plugin 'create mods from an ASI config'. That way there is no need for an ASI  config file which included dos commands like move, robocopy, delete etc.

I have several ideas how you could optimize your standalone ASI further and create parts of the config automatically (with regard nexusid, category, modname) but I'm not sure that would be in your interest in the long run. As in save your sanity. ASI will keep getting bigger and the users want an UI to be able to install only a part of S.T.E.P. 2.27, want a list of mods they have to download (taken into account what they already have installed/downloaded) etc etc. All kind of stuff MO already does or has support for.

My advice: Use the knowledge gained, use the new insights which have surfaced (maintenance issue) and start fresh by creating  a plugin for MO. I know first hand how frustrating it can be to have hit a wall and how irritating  it can be to receive 'advice' which basically tells you 'close but no cigar'. The truth is, sometimes starting fresh is the best and only way to go. I wish I could do that in RL but managers don't like that and  I often have to 'enhance' a program which you should not enhance. Costing more money (and blood sweat and tears and swe*aring) in the long run then doing it right and start fresh. Managers never learn..........

That said, should you want to continue with ASI as an standalone executable I will try my best to help you further.

EDIT: Noticed that Z had created a post when I was creating mine. He basically is saying the same as I am.
  • 0

#138 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 20 November 2013 - 04:09 PM

All of the power of ASI, in its current form, is in its ability to perform script like functions on individual mod archives. In STEP 2.2.7 there are many mods that have file specific instructions like "delete this folder", "delete these files", ect, that not even MO can automate, and is left to the individual to manually operate on their mod archives. These are the parts of the STEP install that needs automating the most, and, with the current architecture of ASI, MO, and NMM for that matter, this is the only way to implement that kind of functionality. It could be done differently with a much more technical algorithm (that was what Monty was working on), but I don't know how to do it and its probably beyond the abilities of one person to implement in a short time, by themselves, anyway. Integration with MO is where I want to take this. There is a tremendous amount of value in using MO's best features; uninstall individual mods and update individual mods. Absolutely an ideal ASI should be able to provide that kind of power and flexibility to the user. But that won't make maintaining it any easier. Based on my current understanding of how MO works I don't think a "from scratch" approach is the best way to do it. The goal here is to make it so that the user has to press 1 button and the installation happens automatically, hands free. MO can't do that, correct me if I am wrong. Even if ASI could unpack mods from within MO as if MO did it by itself, that doesn't do any of the low level stuff I mentioned before, and the user would still have to navigate through FOMOD menus. If all ASI does is feed mods into MO automatically as if the user loaded them up 1 by 1, then ASI isn't really worth doing. A person can already do that by just downloading them from the nexus into MO directly, why have a middle man that, at best, does the exact same thing? The goal of an automated installation is to lower the barrier of entry for newbie modders. STEP does all this wonderful work creating these stable, cohesive mod packages, that only individual with the time and know-how can use. Z you have said before that this is the way it should be, that if you want all that content you have to pay for it by learning how to do all of it. I reject that kind of elitism. Forgive me for getting philosophical here, but the reason we bother to do any of this stuff is because we love skyrim and the modding community, and we want as many people as possible to share in that fun with us. We work hard for that goal, for the person that doesn't know how to do it on their own and needs our help. Its not only that ASI is a solution to that problem, its the best solution to that problem. I don't want to create a tool that makes modding skyrim a little bit easier for the people already doing (and would it really even do that?). I want to make it possible for someone who would never mod otherwise be able to do it because of ASI. So having said that, I have to make it clear that I no longer think an export function for a mod manager is the right approach and its not something I plan to work on. Not because MO isn't a really good app, but because that wouldn't be powerful enough to do the things I think ASI should do. I am open to knew ideas. There might be a better way to do it. But I don't want to stray from the idea that ASI should be able to do everything by itself, not the easy parts, but everything.
  • 0

#139 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,313 posts

Posted 20 November 2013 - 06:07 PM

In its simplest form, we ultimately need a tool that will scan a directory representing a full Pack or STEP install and use this CRC list to identify corresponding CRCs within a group of archives (or directories representing archives) to then repackage said files within said archives (or directories representing archives) as a new package(s) representing a complete mirrored install (or something close, depending on the user's archive complement) which can then be treated as a comprehensive package. This is essentially what mod managers do, but it relieves the user of the painstaking process of configuring each mod and installing them independently in the proper order. This is maintainable by our team (and Pack authors) in that we simply must perform the "directory install" of all relevant files in the proper sequence, which is a prerequisite of even playing a modded game, which in turn is something that most guide creators will be doing anyway. This is how/why it is maintainable. As I have said before, this is exactly analogous to the construction of a BCF in Wrye. One identifies source package(s) and the destination result, and Wrye computes a diff as a BCF. The user simply applies this BCF and points to the input package(s), and the result is a configured package that will yield a mirror of the original "directory install" that any mod manager can handle. We just provide a list of links/mods for the user to download and place into a "source" directory adjacent to the "BCF". We can essentially do this right now using Wrye Bash, but that tool is not simple for novice users, and it is buggy in some regards. It also requires an exact match of input mods and breaks quite easily. This could all then be enhanced with methods like wolverine describes and incorporated as a plugin into MO.

#140 wolverine2710

wolverine2710

    Mod Author

  • Contributors
  • PipPipPipPipPipPip
  • 441 posts

Posted 20 November 2013 - 06:31 PM

@mothergoose729. Programmers are just like normal people, they can develop tunnel vision and are succeptible to the golden hammer  anti-pattern (google both). This can happen to me, Z and also to you. All roads lead to rome, its a matter of finding the fastest, shortest, most efficient path to rome. In this case the path which is best maintainable for now and in the future.  For programmers (at least for me) things tend to be either black or white which is not good when debating things. Perhaps you have misunderstood me. Please reread my post #88 and #107. In MO the packager manually does what needs to be done, install mods, delete files from them, move them around etc etc. Then an XML exporter file is created by MO or an external program which contains the exact contents of all mods in MO. Then an ASI importer plugin for MO or an external program reads in this file. Using a MD5/CRC signature files inside a fomod installer can be located and copied to the correct place. That said there is also post #83 from Tannin to consider. Perhaps I suffer from tunnel vison and is MO is my personal 'golden hammer'.  I'm going to bed and think it all over.  If/when I think I can contribute something meaningfull to this discussion I will report back. I hope others can reread this whole thread and come up with pros and cons for an external ASI vs MO based ASI solution. Perhaps they have brilliant ideas to share with us and solutins we didn't think about. But the most important thing is MAINTENANCE for S.T.E.P. and for common/casual users which want to share their particular setting with a friend. Users who don't know nothing about a CLI (command line interface), dos command like move, delete, robocopy. They needs a tool where they press button 1 which generates 'something'. They give this 'something' to a friend. This friend presses button 2 and all of the sudden he/she has the same setup as his/her friend. THAT IS THE MOST IMPORTANT THING. Edit: Z has this nice trait/characteric of typing text at the same time as I do it BUT he's much faster then I am. So again my post is heavily overlapped by Z'sd one.
  • 0

#141 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 20 November 2013 - 06:32 PM

I don't want to say no, but its complicated, like REALLY complicated and difficult stuff, that I have absolutely no experience with. The task of scanning lots of files in lot of different places to get their checksum keys, and then recursively searching through mod archives to find the same checksum, and then recreating that same file structure is a daunting task. The hardest part is knowing what to do when you find a match. When you find a file and your like "aha, you are what I am looking for!" then you have to put that file where it belongs, which is actually much more difficult than finding what you were looking for to begin with. Could I do it... don't know honestly. I kind of want to try. But for certain I couldn't commit to a project like that until this summer. The current version of ASI is the best I can deliver in the near future, unless tannin or somebody wants to hold my hand and explain it to me so I can write something better (not likely). EDIT: Wolverine, the hardest part about using a key, md5 or crc or whatever, to identify files is putting them back where they belong. An XML document that prints the keys in the destination is a great start, but it would take a lot of effort to find the keys in the source and move them where they belong. Unless I am missing something.
  • 0

#142 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,313 posts

Posted 20 November 2013 - 08:47 PM

That is why I am pointing to Wrye Bash. The code to read CRCs and construct archives from these and their path information is already written. It just needs someone that can interpret the methodology and port it into another environment with a few tweaks. It is written in Python.

I would love to do it, but I am only an aspiring programmer with no real talent and so much other stuff to do around here :/

The python code for dealing with BCFs is in bosh.py in Wrye Bash (Mopy/bash/bosh.py) beginning around line 8000 (search for 'bcf'). This may be instructive at least for learning how to access the CRCs and use them and the path info to construct/deconstruct file hierarchy within archives.

#143 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 21 November 2013 - 02:26 AM

I hate to sound like a broken record, but what you're describing is essentially what my effort was doing - all the code you need for scanning hashes and directory structures and replicating them on a target PC is right there for the taking.
  • 0

#144 z929669

z929669

    Ixian Inventor

  • Administrators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 9,313 posts

Posted 21 November 2013 - 03:20 AM

[quote name=''MontyMM' pid='58597' dateline='1385018794']I hate to sound like a broken record' date=' but what you're describing is essentially what my effort was doing - all the code you need for scanning hashes and directory structures and replicating them on a target PC is right there for the taking.[/quote']
That is what I had thought, but it seems that MG et al are not interested in using those BAT scripts, so I thought that they might be interested in using Python. It all seems like duplicated effort nevertheless.

Could ModMerge be wrapped up into a plugin for MO?

#145 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 21 November 2013 - 03:55 AM

Bear in mind that the batch scripts were just a quick and dirty way of implementing some simple steps on top of what Skyrep can do. If you dig into the source code, all that hashing and directory structuring power is in there.
  • 0

#146 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 21 November 2013 - 10:51 AM

I will check it out monty. I was thinking about the problem today, and I have some ideas for how the processing can be done. The keys could be hashed into a hash table. The items in the table would be objects holding the relative path, file name, and exact checksum keys. If I could get far enough that I could get all the keys in the table the rest would be really easy. Ill look into it. I know skyrep has a lot of pieces. Ill try and figure it out.
  • 0

#147 wolverine2710

wolverine2710

    Mod Author

  • Contributors
  • PipPipPipPipPipPip
  • 441 posts

Posted 21 November 2013 - 11:51 AM

@mothergoose729. Noticed that when I was finished with the post you already had responded. You ALL seem to do that..
So a lot will obsolete but I dislike throwing away posts, so here it comes.

Lets try it from a different angle. Lets investigate the complexity of the code and what you need to do. Lets start with what you have tagged as the hardests part: Finding of files with a specific MD5/CRC inside a mod (zipfile). The trick is to use libraries which can do the hard work for you so you can concentrate on the fun stuff, ie ASI.

MD5/CRC handling.

I've done a bit of digging. sfk gives you what you need: "create list of md5 checksums over all files in a folder.by default, this also includes all subfolders (recursive tree walk).". You could parse the output of that external program but perhaps we can perform that action ourself. What is needed is a C++ library which does the following: "Walk a directory/Recursively". This is part of a C++ library called Boost. Let call the function FileTree.  You point filetree to the directory you want to traverse. Every call to Filetree gives you a different filename. You can calculate the MD5/CRC value for that file. Now you put both filename and MD5 value in a hash table. The 'key' is the MD5 number and the 'value' is the filename. Now finding the filename of a file (in a mod directory) with a certain MD5 value is reduced to doing a lookup of the hash table. Each MD5 value represents a filename.
When processing the ASI exporter XML file (AEX) you process each mod after each other and at the end create a mod in /mods/. For each mod you have info like name of the zipfile,modname, nexusid, category and a list of files each with a certain MD5 value. Each file is located somewhere in the zipfile. Unzip the modfile, run FileTree and you end up with a hash table. When processing a mod you start with file1, this has a MD5 value of MD5_File1. Do a lookup in the hashtable with key MD5_File1 and you find the place in the unpacked directory. Now you can put that file in the modname mod at location /mods/ in a certain place, ie textures/wolverine.dds.

Iirc Tannin mentioned that the used  archiving library has a way of accessing CRC value inside a zipfile. Perhaps you can use that library to fill the hash table as well. Tannin is also accessing the filing system, so perhaps he knows a better  library.

Creating an ASI exporter file

I'n not an XML expert but have used it in the past.
The ASI exporter creates an XML file which contains all the information about the current mod setup in MO. In the MO UI all this info is known. Not sure if there is atm an API which can be used from a plugin to access all that info. If not iirc Tannin said he would extend MO so that the required info is exposed. Lets assume that API is in place. What you most likely do is to built your own object/structure which hold all needed information. Now an XML file has to be created. There are two ways to do this. First method. Create an XML file with a certain structure as you deem needed. This can be done with an XML parser. Second method: use an XML marshaller. For Java there are plenty and I assume they will also exist for C++. Basically an XML marshaller takes an object and automatically creates an XML structure or file for it. You have less control over the XML layout but it just works.

Reading in an XML exporter file.

When creating mods in MO the ASI plugin reads in an XML file which contains all information. With an XML unmarshaller you point it to an XML file and it automatically creates an object for it which you can access. This is the same object as when the XML file was created. With an XML parser you can also process an XML file but you have to do more yourself. There are quite a few XML parsers. There are two most popular APIs for XML parsers:
  •     DOM - Tree-based parser that reads entire document into the tree structure.
  •     SAX - Event-based parser that operates on each piece of document sequentially.
SAX provides a mechanism for reading data from an XML document that is an alternative to that provided by the Document Object Model (DOM). Where the DOM operates on the document as a whole, SAX parsers operate on each piece of the XML document sequentially. DOM takes up more memory but you have access to all XML tags directly.

Tannin is already using an XML parser for the NMM importer. Perhaps he is even using an XML (un)marshaller. So he can help you out further with additional information.
  • 0

#148 mothergoose729

mothergoose729

    Thane

  • Members
  • PipPipPipPipPipPip
  • 462 posts

Posted 21 November 2013 - 01:27 PM

Hey wolverine thanks for the information. Given me a lot to think about. For my part I will be busy up until about the 16th of December. I have to prioritize my classes over my side projects, and my classes are getting into finals. After that I will be on break for about two weeks. I will give it my best effort for at least that long and see what I can do. If anybody else wants to run ahead with this thing by all means, I will try and catch up when I have the time.
  • 0

#149 MontyMM

MontyMM

    High King

  • Site Founders
  • PipPipPipPipPipPipPipPipPipPipPipPip
  • 1,144 posts

Posted 21 November 2013 - 01:47 PM

I'll have a play if I can. I think my favoured solution at this point would be some blend of Skyreps hashing, and outputting custom profiles to MO.
  • 0

#150 wolverine2710

wolverine2710

    Mod Author

  • Contributors
  • PipPipPipPipPipPip
  • 441 posts

Posted 21 November 2013 - 03:41 PM

Hey wolverine thanks for the information. Given me a lot to think about.

For my part I will be busy up until about the 16th of December. I have to prioritize my classes over my side projects, and my classes are getting into finals. After that I will be on break for about two weeks. I will give it my best effort for at least that long and see what I can do. If anybody else wants to run ahead with this thing by all means, I will try and catch up when I have the time.


Your welcome. I wish you all the luck with your finals which of course takes priority. I have finished the part about the XML exporter. Tannin is already using similar stuff in the NMM importer to read the XML file created by NMM. I know that looking at someone else code and trying to figure out what is going and then use it, it can be very time consuming. Especially if they are not using libraries to make things easier. I hope I was able to explain that using the right libraries you can start fresh with your own code and that it probably is less difficult then you first thought. Of course I've simplified things and things like a good UI will give you some troubles. But remember that the current MO plugins also use an UI, at least the manual installer and the NMM importer so you can have a look there.  If in the future you need more information I will be glad to help you further, but I'm not a c/C++ programmer (Java guy).

I think you (and for that matter S.T.E.P. and we all) have come a long way. It looks as if the blueprint for an Automatic Skyrim Installer which is maintainable by S.T.E.P. staff and also usable for friends who want to share their setup with other is in place. Again success with the finals and enjoy your future coding efforts.
  • 0


0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users