ID:2820967
 
Resolved
Byondapi is now available. This is a set of structures and functions, that can be extended in the future, for greatly improved communication with native code in external libraries. call_ext() can call functions in the new format by prefacing their name with "byond:", which means no more having to convert everything to and from strings. Byondapi is capable of operations like reading and writing vars, calling procs, and more. An example project for your own external libraries is included in the BYOND distribution.
Applies to:DM Language
Status: Resolved (515.1609)

This issue has been resolved.
Continuation from a discussion in the coderbus discord.

The plan is to overhaul the DM FFI system for DLLs, so instead of the expected external args being `(int n, char *v[])`, we could have a hypothetical `(int n, dmobject *v[])`, where dmobject is a Ctype with methods to `get(str)` to get a variable on the DM type, and a `set(str, value)` where value could be an int or a string, I dont expect this API to support refs or similar.


"But how will we know what call interface to use?"

Simple! Make call accept 3 arguments, but still only require 2. This way, existing behaviour is kept, but for new behaviour, you can now just do `call(dll, "function", DLL_CALLTYPE_DMOBJ)`

"But can you already do all of this with JSON encode/decode"

Absolutely yes, but theres an issue with that. Any speed gain you get from using a DLL to do things (which is the primary reason we want a rich object API), is completely wiped out when you have to serialise and deserialise it to and from JSON DM side, which can involve a lot of list accesses based on the amount of args you are passing down, not to mention you have to rely on your other language to have good serialisation and deserialisation. I'll put it this way, if we get decent FFI with a DMOBJ ctype or similar, I will write CUDAmos, dont try me.


Obviously this is to be expanded on, I am just noting stuff down here for organisation sake.
So I have some thoughts on the API side of this. I'm thinking we at least need a class for an external version of Value, that can handle strings, and another class for a flexible array of those values.

So far I think at least four calls (from external to internal) are needed:

Byondlib_GetVar(ExtValue const &ref, char const *varname, ExtValue &result);
Byondlib_SetVar(ExtValue const &ref, char const *varname, ExtValue const &val);
Byondlib_ReadList(ExtValue const &ref, ExtValueList &list);
Byondlib_WriteList(ExtValue const &ref, ExtValueList const &list);


The list read/write routines are for when you have a reference that's a list value, and you want to get at its internals, or assign an array to a list reference. I probably need something for associative list support, but I think these would be pretty useful.
In response to Lummox JR
Lummox JR wrote:
So I have some thoughts on the API side of this. I'm thinking we at least need a class for an external version of Value, that can handle strings, and another class for a flexible array of those values.

So far I think at least four calls (from external to internal) are needed:

> Byondlib_GetVar(ExtValue const &ref, char const *varname, ExtValue &result);
> Byondlib_SetVar(ExtValue const &ref, char const *varname, ExtValue const &val);
> Byondlib_ReadList(ExtValue const &ref, ExtValueList &list);
> Byondlib_WriteList(ExtValue const &ref, ExtValueList const &list);
>


The list read/write routines are for when you have a reference that's a list value, and you want to get at its internals, or assign an array to a list reference. I probably need something for associative list support, but I think these would be pretty useful.

My one concern with this is that of typing, not DM types itself, but Ctypes being awful to deal with.

I assume `ExtValue` here is an outer class which can be either an object or a primitive (string, number, etc), and ExtValueList is akin to a C-array of ExtValue with getters and setters?

If so, these are already incredibly good, and we shouldnt need to ever instance ExtValue manually as that will always be returned from `Byondlib_GetVar()`. Do correct me if I am wrong on anything.
In response to AffectedArc07
Yes, that's basically right.

There are plenty of reasons you'd need to instance ExtValue manually for strings and numbers, though, so there are constructors and = operators for that. There's also a datatype+ref constructor.

In order to store strings it always uses its own unique copy of a string, and the ExtValue class destroys the copy appropriately.
I felt the need to update this thread because of something which arose during the threaded maptick tests that were done on tg.

I developed a library for integrating lua scripting into DM, but it requires the use of another library which uses memory-hacks to access internal BYOND functions for value manipulation and proc calling.

This causes Dream Daemon to crash when a lua script attempts to access/mutate a DM value while it is being read by the maptick thread. Due to this, I doubt this memory-hack-reliant library could ever see use in the future.

I still want lua scripting to remain possible, and the FFI functions described above provide most of the functionality I would need to recreate my library. However, the ability to call procs from within an external library is not included among the proposed functions. If something of the sort was not already planned, I envision a function along the lines of
 ExtValue& Byondlib_CallProc(const ExtValue &ref, char const *procname, ExtValueList &args)


If such a function were not available, I could still use a workaround involving making a lua script yield to DM when asked to call a proc, and have the DM code for the library call that proc and resume the yield with the proc's return value as the resume arguments, but directly calling procs from FFI would be much more convenient (and likely more performant).
However, the ability to call procs from within an external library is not included among the proposed functions. If something of the sort was not already planned, I envision a function along the lines of

To quote from the patreon posts (now public)

"The next part is that I want to have at least four new functions libraries can call to get and set vars, get and set list contents, and possibly run procs. This has to be done on the server thread, so by definition these calls would only be allowed during a call_ext() out to the library."

Lummox JR resolved issue with message:
Byondapi is now available. This is a set of structures and functions, that can be extended in the future, for greatly improved communication with native code in external libraries. call_ext() can call functions in the new format by prefacing their name with "byond:", which means no more having to convert everything to and from strings. Byondapi is capable of operations like reading and writing vars, calling procs, and more. An example project for your own external libraries is included in the BYOND distribution.