Jump to content


Photo

[WIP] xedit-lib -- DLL Wrapper for xEdit

xedit library wrapper dll programming framework

  • Please log in to reply
15 replies to this topic

#1 Mator

Mator

    Jarl

  • Mod Authors
  • PipPipPipPipPipPipPipPipPip
  • 627 posts

Posted 02 November 2016 - 11:23 PM

s4Krc.png

 

xedit-lib

 

 

Description

xedit-lib is a project to build a DLL Wrapper for the xEdit framework.  This will allow developers in the TES/FO modding communities to make use of xEdit's record definitions and functionality to read and write plugin files from the language of their choice.  Developers can then focus on implementing larger solutions rather than constantly re-inventing the wheel.

 

The library will include extensive documentation, a full suite of tests, and plug-and-play wrappers for multiple programming languages.  The languages which I plan on building wrappers for, in order of priority, are listed below:

 

- Delphi (primarily for testing)

- Javascript (with NodeJS)

- C#

- Python

- C++

- Ruby

- Java

 

 

Design

The library will be designed similar to xEdit scripting, though with a number of notable improvements.  A primary goal is to yield enough functionality from the library to create a full GUI application as an alternative to TES5Edit at a future point in time.  It's also important to return arrays from library functions so users can use functional programming paradigms for iteration, filtering, mapping, etc.

 

All functions in the library will use the cdecl calling convention for maximum accessibility.  All functions will use boolean return values to indicate whether or not the function executed correctly.  All functions are internally wrapped in an exception handler which prevents exceptions from bubbling up to the calling application (which results in runtime errors).  The library will allow applications to load multiple load orders without having to unload the DLL.

 

The library will never directly pass out interfaces to files, records, groups, or elements.  Instead, when an interface would be returned from a function the library instead stores the interface in an internal list and passes out an index (or handle) to the stored value.  The calling application can release a single handle, or reset the entire interface store between extremely large operations.  This should not be an issue, as the limit for handles is ~4 billion, which could only be reached if you loaded ~4GB of plugins or stored all elements in all loaded plugins multiple times over.  We would run out of RAM accessible to the 32-bit DLL before we would run out handles.

 

The library will feature extensive serialization methods, which will allow users to serialize and deserialize elements, records groups, or even entire plugins to/from JSON, XML, and YAML.

 

 

Progress

You can track my progress on the library from the GitHub repository.  The planning document shows an overview of the functions that have been implemented and have not been implemented.

 

As of 11/2/2016, I'm close to being halfway done implementing the library itself.  Once the library itself is implemented, I will be working on the testing suite and wrappers for different programming languages.

 

If you are a software developer and would like to help make this project happen sooner, please let me know.  I'd greatly appreciate any help the community can offer me on this project.

 

 

Use Cases

I envision a number of use cases for this project, including patching, automation, resolving plugin errors, and building new mods.  This project is, in many ways, a spring board from which many other projects can be made.

 

I personally plan on making an Electron AngularJS application using this library.  Working on Mod Picker has shown me that I really like the architecture and systems involved in web development, which are by my measure superior to desktop application development in many areas.  (most notably GUI design)

 

Please refer to the use cases document for more information about how I plan on using this library in the future.

 

 

FAQ

Q: What about Mator Smash and Merge Plugins?

A: Those application will continue to be supported and developed.  While fighting with Delphi to get the GUI to act as I want is very tiring at times, the battle is mostly won for these applications.  I do not currently plan on phasing out support for them any time soon, and I am looking forward to coming back to them to polish things off.  It is possible that these applications may be "remastered" into Electron AngularJS applications using xedit-lib sometime in the distant future.

 

Q: What about Mod Picker?

A: Mod Picker is still my primary focus.  I'm only working on xedit-lib in my off-hours.  (~10 hours/week, where 60-90 hours a week goes to Mod Picker)

 

Q: This is awesome, but I'm not a developer, how can I help?

A: I don't have a job right now.  I'm building modding tools full time right now.  I'm hoping to continue doing this for the next 6 months, but when my funds dry up I will, inevitably, have to go out and get a Software Engineering position somewhere locally.  If you'd like, you can donate to me on PayPal to support my efforts.  Additionally, I greatly appreciate your feedback and encouragement on all of my projects.

 

Q: Can you really match or exceed the xEdit GUI's functionality in a new application?

A: Absolutely.  xEdit has a fairly simple GUI, and building a GUI using HTML/CSS/JS is a heckuvalot faster, easier, flexible, and customizable compared to building a GUI with Delphi or another programming language.

 

Q: When will you be done?

A: I expect to have the library "completed" sometime in December.  By Christmas at the latest.  The GUI application will follow, with an alpha release ready for use by 2017.  Things should be pretty solid by the end of February.  This is all just estimations though, and really depends on how much time and work Mod Picker will require over the next few months.


Edited by Mator, 02 November 2016 - 11:39 PM.

  • 2

#2 GrantSP

GrantSP

    The antipodean

  • Super Moderators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,333 posts

Posted 03 November 2016 - 02:26 AM

Sorry to here you're currently unemployed, hope you are able to get some funds come in shortly.

I'm assuming the Delphi side of the things is coded in RAD Studio Delphi XE, what about the GUI, what's your setup for that? I'd love to have a fiddle with that side of things if at all possible.



#3 Mator

Mator

    Jarl

  • Mod Authors
  • PipPipPipPipPipPipPipPipPip
  • 627 posts

Posted 03 November 2016 - 02:48 AM

Sorry to here you're currently unemployed, hope you are able to get some funds come in shortly.

I'm unemployed by choice, no need to apologize!  :P
I am working on Modding-related tools full-time because I love doing it, and the experience I get/projects I complete will allow me to ask for a larger salary in the future.
 

I'm assuming the Delphi side of the things is coded in RAD Studio Delphi XE, what about the GUI, what's your setup for that? I'd love to have a fiddle with that side of things if at all possible.

Well this project is a library.  The GUI of the (planned) application (name TBD) will be in HTML, CSS, and JavaScript, leveraging AngularJS and Electron.


Edited by Mator, 03 November 2016 - 02:48 AM.

  • 0

#4 sheson

sheson

    Dragon King

  • Mod Authors
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,748 posts

Posted 03 November 2016 - 05:35 AM

 I foresee a stable platform independent CK as a service in a browser.


Edited by sheson, 03 November 2016 - 05:38 AM.

  • 0

#5 dunc001

dunc001

    Thane

  • Members
  • PipPipPipPipPipPip
  • 466 posts

Posted 03 November 2016 - 08:54 AM

Awesome work.  Just sent you some beer and pizza to keep you ticking :D

 

Confirmation number: 5FJ48912ST116524N


  • 0

#6 Mator

Mator

    Jarl

  • Mod Authors
  • PipPipPipPipPipPipPipPipPip
  • 627 posts

Posted 03 November 2016 - 12:40 PM

 I foresee a stable platform independent CK as a service in a browser.

It's actually possible, though definitely a ways off.

I mean, if you can do things like this: https://www.3dtin.com/ in a browser, then you can do anything.

 

Awesome work.  Just sent you some beer and pizza to keep you ticking :D

 

Confirmation number: 5FJ48912ST116524N

Thanks man!  :D


Edited by Mator, 03 November 2016 - 12:40 PM.

  • 1

#7 Kesta

Kesta

    Thane

  • Mod Authors
  • PipPipPipPipPipPip
  • 477 posts

Posted 03 November 2016 - 05:23 PM

I am exited. Aroused even. Way more than for most of your other projects (and you know how I felt about them :P)

 

 

I wish I had more time, but for now, anything modding-wise that isn't "Special Edition Emergency" go after work on Beyond Skyrim for me. Depending on when you start work on the C# wrapper, I might lend a hand (though I'm expecting several people might help a lot on this one as well... ).

 

Also probably going to rewrite a GUI based version of Anwynn whenever everything is complete and I have time :) 


  • 0

#8 GrantSP

GrantSP

    The antipodean

  • Super Moderators
  • PipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPipPip
  • 4,333 posts

Posted 03 November 2016 - 05:26 PM

I'm unemployed by choice, no need to apologize!  :P
I am working on Modding-related tools full-time because I love doing it, and the experience I get/projects I complete will allow me to ask for a larger salary in the future.
 

Well this project is a library.  The GUI of the (planned) application (name TBD) will be in HTML, CSS, and JavaScript, leveraging AngularJS and Electron.

Thank you.

Electron I'm familiar with as I have been using Atom for some time now. Here's hoping this/these projects provide you with all the experience you need to get what you want.



#9 Mator

Mator

    Jarl

  • Mod Authors
  • PipPipPipPipPipPipPipPipPip
  • 627 posts

Posted 03 November 2016 - 06:57 PM

I am exited. Aroused even. Way more than for most of your other projects (and you know how I felt about them :P)

 

I wish I had more time, but for now, anything modding-wise that isn't "Special Edition Emergency" go after work on Beyond Skyrim for me. Depending on when you start work on the C# wrapper, I might lend a hand (though I'm expecting several people might help a lot on this one as well... ).

 

Also probably going to rewrite a GUI based version of Anwynn whenever everything is complete and I have time :) 

Well, it is a project which yields an API for a large number of other projects (in many languages), so it's pretty meta.  ( ͡° ͜ʖ ͡°)

 

Sure, anything you can offer would be awesome.  I don't expect to be working on the C# wrapper until early December, so you have time.  ;)

 

Thank you.

Electron I'm familiar with as I have been using Atom for some time now. Here's hoping this/these projects provide you with all the experience you need to get what you want.

Yeah.  Between this and Mod Picker I think I will have the experience to get any local software development position (except government) with a salary of $100K or more.  Maybe I'm exaggerating/dreaming, but when I look at other developers my age I feel that I have a big leg-up due to the quantity and quality of the projects that I've done.


  • 0

#10 Mator

Mator

    Jarl

  • Mod Authors
  • PipPipPipPipPipPipPipPipPip
  • 627 posts

Posted 26 November 2016 - 10:39 PM

Spent some time working on xedit-lib today, still doing more but the progress so far is looking pretty good:

 

 

Test results generated from this code: https://github.com/m.../DelphiTest.dpr


Edited by Mator, 26 November 2016 - 11:15 PM.

  • 0

#11 Mator

Mator

    Jarl

  • Mod Authors
  • PipPipPipPipPipPipPipPipPip
  • 627 posts

Posted 27 November 2016 - 10:57 PM

Today I built a testing framework for Delphi taking inspiration from Jasmine.  The framework is called Mahogany, and is available on GitHub.
 
I'll be using this framework to build and perform rigorous tests on xedit-lib functionality.  Because testing code is a good thing.


  • 0

#12 Kesta

Kesta

    Thane

  • Mod Authors
  • PipPipPipPipPipPip
  • 477 posts

Posted 10 December 2016 - 05:30 AM

I started to look a bit more seriously into this.

 

2 questions / remarks

 

In xeMasters, the GetMaster take an index argument as Integer:

 

1) I believe this should be specified as an unsigned value (either with those fancy cardinal, or something smaller like whatever SByte are in delphi... "ShortInt" by the look of things ?).

 

2) As for the "Integer" type itself... I've read something on a freepascal wiki page that "Integer" in pascal could be either 16 bits or 32 bits. Though the "official" (?) Delphi documentation state it's 32 bit without any comment on the subject... any insight on this ? 

 

 

 

 

Edit: Ah, and while I think about it... do you intend do distribute it as is with an installer and have the lib in the PATH so various tool can access it easily, or rather have each tool shipping with the lib ? (or try to discover it on its own ?)

 

Edit2: And one more questions about your naming conventions: what do the underscore mean for arguments like "_id" ? 


Edited by Kesta, 10 December 2016 - 05:45 AM.

  • 0

#13 Mator

Mator

    Jarl

  • Mod Authors
  • PipPipPipPipPipPipPipPipPip
  • 627 posts

Posted 17 December 2016 - 07:31 PM

I started to look a bit more seriously into this.

 

2 questions / remarks

 

In xeMasters, the GetMaster take an index argument as Integer:

 

1) I believe this should be specified as an unsigned value (either with those fancy cardinal, or something smaller like whatever SByte are in delphi... "ShortInt" by the look of things ?).

 

2) As for the "Integer" type itself... I've read something on a freepascal wiki page that "Integer" in pascal could be either 16 bits or 32 bits. Though the "official" (?) Delphi documentation state it's 32 bit without any comment on the subject... any insight on this ? 

 

Edit: Ah, and while I think about it... do you intend do distribute it as is with an installer and have the lib in the PATH so various tool can access it easily, or rather have each tool shipping with the lib ? (or try to discover it on its own ?)

 

Edit2: And one more questions about your naming conventions: what do the underscore mean for arguments like "_id" ? 

 

Hi Kesta!  I'll try to answer these:

 

1) Sure, that'd be fine.

 

2) It should be 32-bit on all modern operating systems.  I think it was 16-bit on 16-bit systems (back when those were a thing).  :)

 

 

I think putting the library on the user's path would be good.  Developers who use the library heavily are free to package it with their application as well.

 

I prepend an underscore on variables which store values which are only meaningful within the context of the library.

  • I use it for variables that are "handles" to interfaces as defined by the library: "_id" and "_res". (res being short for result)
  • I also use it for the internal handle/interface data structures: "_store" and "_releasedIDs".
  • I also use it for IwbFile variables as "_file" because "file" is a reserved word in Pascal.

Edited by Mator, 17 December 2016 - 07:44 PM.

  • 0

#14 Mator

Mator

    Jarl

  • Mod Authors
  • PipPipPipPipPipPipPipPipPip
  • 627 posts

Posted 20 December 2016 - 06:32 AM

I've got some basic plugin file JSON serialization working.  I don't have group/record serialization perfected yet, but it's just around the corner.

 

Skyrim.esm.json


  • 0

#15 Mator

Mator

    Jarl

  • Mod Authors
  • PipPipPipPipPipPipPipPipPip
  • 627 posts

Posted 21 December 2016 - 06:35 PM

I've been making some good progress on building tests with Mahogany.  Man is it nice to make tests with a proper testing framework!

 

Current test output:

 

I've been streaming development on my livecoding.tv channel the last few days.  Feel free to drop by.  :)


  • 0



Also tagged with one or more of these keywords: xedit, library, wrapper, dll, programming, framework

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users