ID:1481007
 

I Hate C++, or why I made call()() work with .NET DLLs



1. I Hate C++, or why I made call()() work with .NET DLLs
2. Why .NET?
3. So how's it work, smartass?
3.1 Strings
3.3 Lists
3.4 Complex Objects
3.5 I skipped a number
3.6 Images
4. Setting up UnmanagedExports
5. Create the DLL
5.1 Set up the project
5.2 Write your function
5.3 Things to know
6. Showtime!
7. Additional Reading
7.1 Other Examples
7.11 File Info
7.12 Concat List
8. Version

I Hate C++, or why I made call()() work with .NET DLLs

If you've ever looked at how call()() works, you might've noticed that there is a nice little function that allows you to call code residing in a DLL. You might've also noticed that it's a rather unfortunately-designed signature that seemingly forces you to use C or C++, and handle only strings, never numbers or more-advanced types.

Well, neither of those is true. How, you ask? Through the magic of computing and taking a third option.

Why .NET?

Because I hate C++ with a fiery passion hotter than the brightest suns.

Okay.. that might be overstating it a bit. C++ isn't bad, but I don't like using it. Give me C# and the .NET CLR any day. Personally, I'm quite happy programming with C#, the classes and functions it gives me are great, and it's very easy. No stupid typecasting problems to deal with, unless you're doing something crazy. Take into account the built-in memory management and JIT compiling, and people are happy campers.

Another reason why is the vast flexibility of .NET, and the number of languages that you can code with. C# and VB.NET spring to mind, both of which can be used here very easily (for my sake, I'm only going to cover C# - VB is the same, except that the syntactic sugar on the attributes changes). Don't even get me started on things like IronPython or F#, the stuff you could potentially do with that boggles the mind! If you're familiar with .NET programming, you'll know what I'm talking about. So.. Ease of use, flexibility, processing power.. I haven't looked into trying to use it yet, but threading's a possibility, too.

So how's it work, smartass?

Well, there was no need to call me that...

The trick is simple. It's possible, by fiddling with the MSIL in .NET assemblies and recompiling them, to create stub functions that when called load the .NET runtime and execute the code you've written. Normally this would take a lot of technical know-how and familiarity with the .NET toolset, but as it turns out, someone else has created an amazing little package called 'UnmanagedExports' that can do almost all of your work for you - all you need to do is set it up and then write code with a few changes made to accomodate it. We'll be covering both of those in later chapters.

In summary: It works because of ~Wizardry~ and technical voodoo. Also, disassembling MSIL bytecode, changing it, and reassembling afterwards. But you don't need to worry about that. It gets done for you.

Of course, there's still that fun little issue of only strings being supported by call()() - but I bet you'd much rather be able to pass objects, numbers, lists, and more into it. Well, that's possible. You have to do a little marshalling on each side, but it's less complex that you might need to worry. See, BYOND has this little feature called "savefiles", and at their core they operate as text files. That means -all- data has to, ultimately, get stored as textual information (with the exception of files and icons - those work a bit differently). This is huge for us, as it means all we have to do is serialize our data to text, pass it through call(), and deserialize on the other side!

In the next few subsections, I'm going to discuss how to marshal each of varying types through call()(), so that you can use them in your .NET code.

Strings

Bet'cha didn't expect this one. Well, there's not much you need to do here. Strings will pass between BYOND and your DLL almost painlessly. In fact, that's the only thing you can pass!. The only hurdle is to get the strings on your .NET side.

The key is that in order to handle the string array BYOND gives us, in .NET you have to break out of the safe, "managed" environment for a bit. In the function call, you aren't passed a string array. You're passed a pointer to an array of more pointers. These pointers point to your actual strings! So, knowing how many strings you have, and having a pointer to the initial 'pointer table' for your strings, we'll use some boilerplate code to get your strings.

Remember that we're not only given a pointer to our argument table (I'll call it "ArgPtr"), but also a variable (I'll call it "NumArgs") that tells us how many entries are in that table. We'll cover where these are defined later, in section 5.

String[] Args = new String[NumArgs];
IntPtr[] ArgPtrs = new IntPtr[NumArgs];
Marshal.Copy(ArgPtr, ArgPtrs, 0, NumArgs);
for(int X = 0; X < NumArgs; X++)
Args[X] = Marshal.PtrToStringAnsi(ArgPtrs[X]);

PtrToStringAnsi is important. There are several ways you can define a string, BYOND uses null-terminated ANSI strings; .NET does not. PtrToStringAnsi takes a pointer (the "IntPtr") to a null-terminated ANSI string, and returns a string that .NET can use. You don't need to understand much; outside of the boilerplate code we'll cover in this and the next subsections, things get a lot easier. After this, you can just use the Args[] array, and not have to worry about anything at all for the rest of your function. Even returning a string is just "return MyString". Nice, huh? Well... not quite.

Lists


Once you start trying to send something that isn't a string, things get harder, and more steps are involved. We'll start with the easiest complex object to send: a list. Let's start by defining a simple list that'll give us something to work with.
var/list/MyList = list("name"="Texas Rex", "dialogue"="I think you need to step aside, compadre.")

Awesome, we have our list. Now, how do we get that to the DLL? Well, that's actually pretty easy. We'll use list2params(), which will take our list.. and convert it to a convenient string! This means we can just pass it directly, without any other work on the BYOND side of things! Now, how do we convert that back into a list? Well, .NET has a handy function for that.

In the System.Web namespace, there is a class named 'HttpUtility'. This class has a function named 'ParseQueryString'. Well, guess what? list2params()... outputs a query string. Seriously. .NET actually has a built-in function analogous to params2list()! So, how do you use it? Let's say you passed only one argument to call()(); list2params(MyList). Assuming you haven't erased the boilerplate code from above:

NameValueCollection MyList = HttpUtility.ParseQueryString(Args[0]);
String MyString = myList["name"] + " says: " + MyList["dialogue"] // MyString = "Texas Rex says: I think you need to step aside, compadre."


Nice, huh? But, that's only associative arrays! What if we used var/list/MyShoppingList = list("Apples", "Butter", "Tex Mex Cheese", "Weapons-grade Plutonium")? Well, .NET has the answer for that, too!

NameValueCollection MyShoppingList = HttpUtility.ParseQueryString(Args[0]);
List<String> MyStringList = new List<String>((IEnumerable<String>)MyShoppingList.Keys); // I highly recommend you grab a good C# book.
Boolean DoINeedApples = MyStringList.Contains("Apples"); // DoINeedApples = True


How's that for you? .NET has a ton of functions and operators; all of your basic list operators and more are present. But.. how do you convert that back into a string? Does .NET have anything for that? Well.. yes, it does.

NameValueCollection MyList = HttpUtility.ParseQueryString(String.Empty);
MyList["DidIBuyApples"] = "Yes";
MyList["DidIBuyThePlutonium"] = "No";
String QueryString = MyList.ToString(); // QueryString = "DidIBuyApples=Yes&DidIBuyThePlutonium=No"


Woo! Now you can return that, and send your list back to BYOND. From there, it's a quick params2list() and you're done - you have a list you made in .NET, back in BYOND. Good job there, compadre!

Complex Objects

Complex objects are things like /datum, /atom, or /obj/item/WeaponsGradePlutonium. They fit into BYOND's tree structure, and you might be able to find them on your map. So, what if you want to serialize your object and send that to .NET? Well, it's doable - but with a caveat. I don't know the BYOND savefile format well enough, and there definitely isn't a .NET class for that. If you want to handle complex objects, you're going to have to write your own parser for the format. So, how do you serialize a BYOND object to a string? Well, if you want to use the utmost minimum amount of code...
var/savefile/S = new()
S << MyObject
var/SerializedObject = S.ExportText()
This will result in SerializedObject containing a text string that you can pass to your .NET code.

Since there's no current way to convert a BYOND object into something nice and pretty that .NET can use, there's not much point trying to explain how to use it - unfortunately, for the time being you're on your own here. IF you're trying to write a parser, I would advise looking into the .NET 4.0 DLR, and dynamic objects.

But let's say you got all that done, and your .NET function has returned a nice, happy object in text form. How do you deserialize that back into a native object? Easy!

var/SerializedObject = call("MyDLL.dll","MyFunc")()
var/savefile/S = new()
var/obj/MyObject
S.ImportText(SerializedObject)
S >> MyObject


And that's how you do it. Keep in mind, this will NOT serialize/deserialize tmp or const vars, only regular ones.

I think I skipped a number

Oh, so I did. Funny, 'cuz this is probably most important to you. How do you serialize and deserialize numbers? Just convert them to strings. Use
var/Text = "[MyNumericVar]"
to serialize, and
MyNumericVar = text2num(Text) || 0

to get your number back as a number. Numbers are really simple, though if you're dealing with variables that can be numbers at some time, and strings at others, you'll have to work a bit harder to get it "right" for you.

Images

Images, as it turns out, are also pretty easy to import/export. You effectively do the same thing as you do with complex objects, but instead of an object graph, you'll get a base64-encoded version of the file. You can then convert this base64 string to binary and then to a bitmap within your .NET program.

Setting up UnmanagedExports

Time to get into the meat of this. I'm going to assume you already have Visual Studio installed. You'll need at least Visual Studio 2010; VS2008 will not work. VS2012 and above should work as well. I'm also going to assume you have a base familiarity with Visual Studio - if not, there are plenty of great introductory tutorials to the IDE.

First, you will need to install NuGet and then restart visual studio. Once you have NuGet installed, you will need to create a solution. Make sure that you select to build a Class Library in the language of your choice; for this tutorial I'm going to use C#. Let Visual Studio set up your environment and project. Now, the fun part. In the command bar, select Tools->Library Package Manager->Manage NuGet Packages for Solution. Click the 'Online' header, and in the search bar enter "UnmanagedExports". Install that, and set it up for your project.

Now our ball is rollin'! After NuGet has set up UnmanagedExports, you're ready to code!

Create your DLL

Okay, now we're done the chit-chat, let's make something good! For this example we're just going to remake the merge() example the help file gives us. I'll split this into two parts: setting up the project, and writing the code.

Set up the project

There is some little setup we have to do with the project. First, in the command bar select Build->Configuration Manager. Click the "Active Solution Platform" dropdown, and select "". In "Type or select the new platform", pick x86. Do not pick x64! BYOND is 32-bit, and if you pick x64 your DLL won't load! Once you have this set up, select x86 as your platform in your build configuration (you can find this in Project->(Name) Properties..., and pick 'Build' on the sidebar in the pane that opens up).

Lastly, we need to add a reference to System.Web, so that we can use HttpUtility. Once this is done, you're ready to write your function. To do this, click Project->Add Reference... in the command bar. Find System.Web, and add it. Now you're done, and ready to write code!

Write the Code!

Don't lie, this is what you've been waiting for the entire time, right? Well, I hope so, because it's the part I've been waiting for!

We already know we want to write merge(). We also have our nice boilerplate code to grab the string arguments. But what about the function itself? How do we link BYOND and .NET? Well, that's where the magic happens.

UnmanagedExports is the key here; it gives us an attribute to play with that lets us define which functions get their nice little export. This attribute is used in the function definition, along with a marshalling function. Let's take a look at it.

using RGiesecke.DllExport;
using System;
using System.Runtime.InteropServices;
using System.Text;

class Class1 {
[DllExport("merge", CallingConvention = CallingConvention.Cdecl)] // Name our exported function, and the Calling Convention is technical stuff.
[return: MarshalAs(UnmanagedType.LPStr)] // Ensure we return a string formatted so BYOND can use it
public static String Merge(int NumArgs, IntPtr ArgPtr) { // BYOND passes us two params, int and char**, so use these.

}
}


This is really all it takes to define a function. Literally all you will ever need to change here is your function name, and what it's exported as. Everything else should ALWAYS stay the same, otherwise you'll break stuff, and probably pretty badly too.

Now, let's pair that with the other boilerplate code we did up earlier.

using RGiesecke.DllExport;
using System;
using System.Runtime.InteropServices;

class Class1 {
[DllExport("merge", CallingConvention = CallingConvention.Cdecl)]
[return: MarshalAs(UnmanagedType.LPStr)]
public static String Merge(int NumArgs, IntPtr ArgPtr) {

// Convert the IntPtr (char** pointer) to a string array
String[] Args = new String[NumArgs];
IntPtr[] ArgPtrs = new IntPtr[NumArgs];
Marshal.Copy(ArgPtr, ArgPtrs, 0, NumArgs);
for(int X = 0; X < NumArgs; X++)
Args[X] = Marshal.PtrToStringAnsi(ArgPtrs[X]);
}
}


Now we're getting somewhere! We've got our function, we've got our string-getter... Let's write the important part of our function, the part that actually puts all the strings together.

using RGiesecke.DllExport;
using System;
using System.Runtime.InteropServices;
using System.Text;

// Lots of boilerplate code
class Class1 {
[DllExport("merge", CallingConvention = CallingConvention.Cdecl)]
[return: MarshalAs(UnmanagedType.LPStr)]
public static String Merge(int NumArgs, IntPtr ArgPtr) {
String[] Args = new String[NumArgs];
IntPtr[] ArgPtrs = new IntPtr[NumArgs];
Marshal.Copy(ArgPtr, ArgPtrs, 0, NumArgs);
for(int X = 0; X < NumArgs; X++)
Args[X] = Marshal.PtrToStringAnsi(ArgPtrs[X]);

// ---------

// The actual code for our merge() function

StringBuilder FullString = new StringBuilder(); // We'll use a StringBuilder class, since it makes the next part easy
foreach(String Arg in Args)
FullString.Append(Arg); // Now build the string...
return FullString.ToString(); // ...and return it!

// ---------
}
}


There. We did it. If you were to take this into BYOND and run it, you'd get back whatever parameters you passed to it. Awesome, huh? Now, you might notice that I could really optimize this function and yes, you're right. I'll leave it as an exercise to the reader however, as I wanted to make the different parts of the function clear.

Things to know

There are a couple of caveats to know. First off is something really useful. If you have any static variables in your class, then they aren't erased between calls to your DLL's functions. This means you can persist state information between calls, as long as dream seeker (or dream daemon) is not closed.

But there's a downside. If something in your DLL can't release itself, then when the server crashes or shuts down, it might actually stay running in the background! I especially ran into this when using try/catch blocks and messagebox.show() for error checking; so be aware of that!

On the topic of Message Boxes, using them does require a little work. First, you'll need to add a reference to System.Windows.Forms, just like you added a reference to System.Web. Then, it's a single line of code:

System.Windows.Forms.MessageBox.Show("Hello, World!");


You could also use the following, if you plan on using Message Boxes a lot:

using System.Windows.Forms;
...
MessageBox.Show("Hello Again, World!");


Showtime!

Well, we made merge. What now? I'll give you a hint: anything you want. Having ready access to .NET with little difficulty means the things you can do with BYOND are going to skyrocket. Use your imagination, and don't be afraid to try new things! The worst you can do is crash your game ;P

Additional Reading

For those of you who want to delve into this further, the best I can recommend is that you go out and check various C#/.NET tutorials. If you're looking for a physical book, I absolutely recommend C# 5.0 Unleashed. This book will take you from the very basics (as if you didn't know what .NET was) and will carry you through C# all the way to advanced topics such as threading, LINQ, and the Dynamic Language Runtime. I have the 4.0 version of this book, and it's truly excellent.

Other Examples


Get file info

This example takes a filename, and will return information about the file. Note that it doesn't handle exceptions though, so hitting an error will crash BYOND, instead of crashing your proc. It also demonstrates building a query string manually, instead of using NameValueCollection.

using System;
using System.IO;
using System.Runtime.InteropServices;
using System.Text;
using System.Web;
using RGiesecke.DllExport;

class Class1
{
private static readonly DateTime UnixEpoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);

[DllExport("fileinfo", CallingConvention = CallingConvention.Cdecl)]
[return: MarshalAs(UnmanagedType.LPStr)]
public static String FileInfo(int NumArgs, IntPtr ArgPtr)
{
String[] Args = new String[NumArgs];
IntPtr[] ArgPtrs = new IntPtr[NumArgs];
Marshal.Copy(ArgPtr, ArgPtrs, 0, NumArgs);
for (int X = 0; X < NumArgs; X++)
Args[X] = Marshal.PtrToStringAnsi(ArgPtrs[X]);

if (NumArgs != 1)
return "";

String Filename = Args[0];

if (!File.Exists(Filename))
{
return "Exists=0&Path=" + HttpUtility.UrlEncode(Filename);
}

FileInfo FI = new FileInfo(Filename);

StringBuilder SB = new StringBuilder();

SB.Append("Exists=1");
SB.AppendFormat("&DirectoryName={0}", HttpUtility.UrlEncode(FI.DirectoryName));
SB.AppendFormat("&Extension={0}", HttpUtility.UrlEncode(FI.Extension));
SB.AppendFormat("&FullName={0}", HttpUtility.UrlEncode(FI.FullName));
SB.AppendFormat("&ReadOnly={0}", FI.IsReadOnly ? "1" : "0");
SB.AppendFormat("&CreationTimestamp={0}", GetUnixTimestamp(FI.CreationTimeUtc).ToString());
SB.AppendFormat("&LastAccessTimestamp={0}", GetUnixTimestamp(FI.LastAccessTimeUtc).ToString());
SB.AppendFormat("&LastWriteTimestamp={0}", GetUnixTimestamp(FI.LastWriteTimeUtc).ToString());
SB.AppendFormat("&Size={0}", FI.Length.ToString());
SB.AppendFormat("&Name={0}", HttpUtility.UrlEncode(FI.Name));
SB.AppendFormat("&Path={0}", HttpUtility.UrlEncode(Filename));

return SB.ToString();
}

public static long GetUnixTimestamp(DateTime TimeRef)
{
return (long)(TimeRef - UnixEpoch).TotalSeconds;
}
}


Concat List

This example takes a list, and concatenates all of the entries (or keys, in the case of an associative array) into a string.

using System;
using System.Runtime.InteropServices;
using System.Text;
using RGiesecke.DllExport;

class Class1
{
private static readonly DateTime UnixEpoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);

[DllExport("concatlist", CallingConvention = CallingConvention.Cdecl)]
[return: MarshalAs(UnmanagedType.LPStr)]
public static String ConcatList(int NumArgs, IntPtr ArgPtr)
{
IntPtr[] ArgPtrs = new IntPtr[NumArgs];
Marshal.Copy(ArgPtr, ArgPtrs, 0, NumArgs);

if (NumArgs == 0)
return ""

List<String> MyList = new List<String((IEnumrable<String>)HttpUtility.ParseQueryString(Marshal.PtrToStringAnsi(ArgPtrs[0])).Keys);

StringBuilder SB = new StringBuilder();

foreach(String S in MyList)
SB.Append(S);

return SB.ToString();
}
}


Version


1.0: Initial Tutorial
1.05: Cleaned up code samples, made things a little clearer
1.1: Added 'Additional Reading' section, cleaned up small typos
1.2: Added image serialization, with many thanks to Hiead.
I'm the exact opposite... C++ is my language of choice and I hate anything Microsoft with a burning, fiery, raining death passion of horror.
It seems kind of misleading to state that you don't have to use C++, because in this case, you are generating stubbs that act as code proxies to your .Net functions.

It also seems kind of misleading to state that you can send datatypes other than strings, when in fact you are sending serialized data in string format, and then deserializing them at the other end.

Solid example, though, semantic nitpicking aside.
Some small updates to the tutorial, to try and improve it a bit.
I'm not a fan of .NET in the slightest, but for completeness I thought I'd give you a helping push for the "Images" section: use the same method you did for "Complex Objects"

client
verb
Test()
var/savefile/s = new
s << 'foo.dmi'
usr << s.ExportText()


. = filedata("name=foo.dmi;length=277;crc32=0x72fc2a03;encoding= base64",{"
BORw0KGgoAAAANSUhEUgAAACAAAAAgAQMAAABJtOi3AAAABlBMVEXAwMD/// 8raYe0AAAAAXRS
TlMAQObYZgAAAGR6VFh0RGVzY3JpcHRpb24AAHicU1ZwcnX39FNw8fXkKkst Ks7Mz1OwVTDRM+Di
ckAcoyNuDgzUjPTM0ognOKSxJJUIFNJiYszJbOoGMg05OJMK0rMTYWwlRVc/ VzAJgIAmrcY
C32rgUMAAABNSURBVAiZY2DgWMCADBrgLJYEIMHRASJAXGYGBwYmBkYgS5jL gYFBACToAJJo4lAA
UZZArIJiFANQnAnKBNMgzRDDoICRAR2woDqmAQD/ qAVZ8Jn4AAAAAABJRU5ErkJggg==
"})


On the DLL end, you just have to parse out the base-64 data. I don't use .NET but I'm sure some library already does this for you.

Worth a shot.
Oh, that's awesome. I wasn't aware that images could be serialized to base64 like that. And yes, .NET does include built-in base64 conversion.