TCL / Shared memory stuff

[Yil]

Here's some info I wrote primarily via PM's to a potential scripter, but figured it might be useful to others so I combined a few replies and cleaned it up a bit...

ioFTPD has a shared memory interface for use by EXEC scripts. It's not really documented but there's some lame examples from the dark ages in source\Window.Messages.And.Shared.Memory that show how to use it a bit. Since ioFTPD is open source grab the 7.7.3 source code and take a look at the DataCopy.[ch] files for the complete list of available commands. You can access all users, groups, vfs settings, list online users, make the server call a script, etc so it's powerful enough for almost all uses. I believe the TCL dll library used by some of the eggdrops also uses shared memory and I think you can find the source for that in the stuff neoxed releases as part of nxtools or maybe it was the dBZbot or Alcobot source or something.

One word of caution with the shared memory interface. Vista+ limits the interaction of different login sessions, and services run in their own session 0 world so I don't think ioFTPD running as a service and ioGUI running in shared memory mode would be able to talk to each other. There are ways around that I think by running the service as a user instead of the system account and enabling the interactive desktop option but I haven't verified that. Remember, this is only an issue for an ioGUI like tool, any normal script will be called from the server and will always be able to talk to the server. ioGUI is in VB6 and has a shared memory interface as well as a plain login interface - source is available for it.

For those C# people, you can't use the shared memory interface in C# directly, what you do is use a small C++ .NET wrapper/dll to do it for you and export that interface to C# so it can be called normally. This is done all the time to get the performance C++ can offer for a variety of features used by C#.

The environmental variables passed to EXEC'd scripts come from the etc\ioftpd.env file. You can add any of the available cookies from doc\Cookies.txt provided they are available (i.e. userfile cookies aren't valid until the user has logged in, etc). Until the upcoming v8 release I don't think there is a "simple" way to accurately get the user's VFS file though. If you have to do it now you can get the one defined in the userfile via %[$VFS], but if not specified you'll need to lookup the VFS path for the user's default group to see if it has one defined, or else examine the ioftpd.ini config file to get the system default VFS file... v8 will make this easier for you as there will be a new cookie with the correct path.

TCL interpretters call scripts/init.itcl before anything else. Don't confuse this with init.tcl which is for TCL itself, you don't want to use that name as it's important since it has to run to setup load paths to default libraries, etc. If you are writing a script that will be called ALL the time, i.e. zipscripts/sitescripts then adding stuff to init.itcl makes sense. If this is a simple standalone site command I wouldn't recommend that. Just define everything in a .itcl and call it since the overhead isn't really that big. You could also do what I do and make one file source another file but only if it hasn't been already done. This gets you the same effect but delays the loading of the options or big TCL script file until it's actually needed which is useful for things that aren't called all the time. Most of the time I don't bother though since you have to 'site rehash' to pickup changes if you do that and development with a single file picks up the changes immediately.

I haven't studied the performance tradeoffs of a fully pre-loaded TCL script versus a C# EXEC script. I'm not sure how much initialization C# has to do before it can print hello (java for instance is really slow but C# is much better), and TCL is slow in general. I might guess that trivial things are actually faster in TCL, but reasonably complex stuff would be faster in C#. I personally think TCL is sorta lame, but any full fledged site/zipscript needs a TCL component for the windrop so learning TCL isn't a bad idea and it might makes the script easier for end users to tweak if they need to.

You can use the DC_FILEINFO_READ and DC_FILEINFO_WRITE shared memory commands to access the vfs data and I believe that give you the permission info as well as the RAW chattr data. The context field of the FileInfo for the chattr stuff is defined in DirectoryCache.h but isn't really described. You'll need to check out the *Context* functions in DirectoryCache.c to figure it out. I didn't design it, but I believe it's a very simple 1 byte context-identifier, 4 byte length field followed by that many bytes of data, and you walk from the start using the length to jump to the next id and look for the chattr id you want (they aren't in any order) and once found just extract the field. To update/insert just remove the old and insert or append the new...

As an aside, it's a 4 byte length field BUT it's not word aligned which doesn't seem to matter, but is technically wrong and you can't have a length greater than 4K anyway at the moment because the routine that writes out chattr data performs checksums on 4k blocks and any single chattr field can't span blocks. So what TCL scripters do it make one big field with the really large data and split the chattr data at 3.8K or whatever and store it in say chattr 160, and then 161, etc and then write out an empty field to indicate it's done and then process that once re-assembled. Most of the time there is only 1 large field like that so it's simple enough to do.

The shared memory interface allows scripts to call back into ioFTPD and execute another TCL/EXEC script to do more work. This is perhaps a useful means of getting iTCL only features available to EXEC scripts, however please let me know if that's the only reason and I'll see if I can't enhance the interface so you don't have to go through all that.

I don't believe there is any current shared memory dll interface that uses the new userfile fields. This means that windrop's can't get things like deletion message, time, etc. It also means that only the first 10 sections are available. Taking the alcobot dll or whatever and upgrading it should be straightforward which would be useful to all TCL windrop scripters. Ripping out all the TCL specific stuff and replacing it with C++ .NET stuff would make a dll that could be loaded from C# that would use the shared memory interface.

I've been focused on using the ordinary FTP login from windrops back to the server to issue commands to TCL scripts and then parsing the results so I haven't really used the shared memory interface myself except to track down some early problems I saw with it way back in early v6 times.

I'm more than happy to help if people have specific questions, need access to information that isn't currently available via the interface, etc.

[jeza]

Having a .dll that can be laoded from c# would be great...

[peep]

Indeed it would. Haven't used shared mem in the past either, but getting the ability to write scripts in C# would be the needed carrot! .NET4 did bring support for some Shared Mem features (memory-mapped files) but can't really grasp the big picture of it..

"The .NET Framework now supports memory-mapped files. You can use memory-mapped files to edit very large files and to create shared memory for interprocess communication."

[jeza]

Something like this would be useful for the start
I am not sure how should the stuff about ONLINEDATA, VFS, KICK look like...

using System.Collections.Generic;

namespace ioFTPDData
{
    public interface IServerData
    {
        /// <summary>
        /// Gets users.
        /// </summary>
        /// <returns>List of users.</returns>
        List<UserFile> GetUsers();

        /// <summary>
        /// Gets groups.
        /// </summary>
        /// <returns>List of groups.</returns>
        List<GroupFile> GetGroups();

        /// <summary>
        /// Gets the spedified user.
        /// </summary>
        /// <param name="uid">The uid.</param>
        /// <returns>User data.</returns>
        UserFile GetUser(int uid);

        /// <summary>
        /// Gets the specified group.
        /// </summary>
        /// <param name="gid">The gid.</param>
        /// <returns>Group data.</returns>
        GroupFile GetGroup(int gid);

        /// <summary>
        /// Adds new user.
        /// </summary>
        /// <param name="userFile">The user file.</param>
        /// <returns><c>UID</c> when user was successfull added, else <c>-1</c>.</returns>
        int UserAdd(UserFile userFile);

        /// <summary>
        /// Updates the specified user.
        /// </summary>
        /// <param name="userFile">The user file.</param>
        /// <returns><c>true</c> when user was successfull updated, else <c>false</c>.</returns>
        bool UserUpdate(UserFile userFile);

        /// <summary>
        /// Removes the specified user.
        /// </summary>
        /// <param name="uid">The uid.</param>
        /// <returns><c>true</c> when user was successfull removed, else <c>false</c>.</returns>
        bool UserDelete(int uid);

        /// <summary>
        /// Add new group.
        /// </summary>
        /// <param name="groupFile">The group file.</param>
        /// <returns><c>GID</c> when group was successfull removed, else <c>-1</c>.</returns>
        bool GroupAdd(GroupFile groupFile);

        /// <summary>
        /// Update specified group.
        /// </summary>
        /// <param name="groupFile">The group file.</param>
        /// <returns><c>true</c> when group was successfull updated, else <c>false</c>.</returns>
        bool GroupUpdate(GroupFile groupFile);

        /// <summary>
        /// Remove specified group.
        /// </summary>
        /// <param name="gid">The gid.</param>
        /// <returns><c>true</c> when group was successfull removed, else <c>false</c>.</returns>
        bool GroupRemove(int gid);

        /// <summary>
        /// Change file or dir owner id or permissions (mode).
        /// </summary>
        /// <param name="mode">File or directory mode. Like 755</param>
        /// <param name="uid">User id</param>
        /// <param name="gid">Group Id</param>
        /// <param name="fileOrPath">File or path.</param>
        /// <example>printf("!vfs:add 777 101:101 D:\\desktop\\ioFTPD\\test\n");</example>
        void VfsAdd(int mode, int uid, int gid, string fileOrPath);

        /// <summary>
        /// Create ioFTPD symlink.
        /// </summary>
        /// <param name="realPath">Real path on disk.</param>
        /// <param name="virtualPath">Virtual path</param>
        void CreateSymlink(string realPath, string virtualPath);

        /// <summary>
        /// Gets directory or file attributes.
        /// </summary>
        /// <param name="fileOrPath">File or path.</param>
        /// <example>Get MODE, UID, GID for the specifed directory or path.</example>
        T GetAttributes(string fileOrPath);

        /// <summary>
        /// Check who is online.
        /// </summary>
        /// <returns>List of online data.</returns>
        List<OnlineData> WhoIsOnline();

        /// <summary>
        /// Kicks user from specified directory.
        /// </summary>
        /// <param name="uid">The user id.</param>
        /// <param name="path">Directory full path.</param>
        /// <returns><c>true</c> on success, else <c>false</c>.</returns>
        bool UserKick(int uid, string path);
    }
}