ioFTPD v6.3.5 Beta Released

[Yil]

Here it is folks. 6.3.0 with a whole bunch of new features and bug fixes!

The most important long term change is that this version should finally make debugging easier for me since it offers 3 ways to provide info on crashes. A text file for the paranoid which is often pretty useful and everyone can send me, a small stack trace which provides lots more data and can be loaded in a debugger to make my life easier, or finally a complete memory image of everything but the executables/libraries for the harder to solve problems.

New features include: Dynamic DNS support in hostmasks, a whole new utility (ioKnock), a rule based system for editing hostmasks so you can control who can set what, better support for immunity from auto-banning, built in IP/Host obscuring in the logfiles which should remove the need for local BNCs, new commands to generate/manage SSL certs automatically, logfile spam protection, etc...

Notable fixes: MDTM, 0 byte file problems, removing "empty" dirs, etc...

Report problems here and if crashing include the logfile. Also remember to PM me for info on how to send the dump files as they are easier to work with.

Latest Version:
ioFTPD-v6.3.5.zip

[Yil]

v6.3.0 Release Notes:

*** File Modifications

1) File system\ioFTPD.exe changed. Version 6.3.0.0

2) File system\ioFTPD.pdb added.   Debugging info. v6.3.0.0

3) File system\tcl84t.dll changed. Upgraded tcl to version 8.4.15.

4) File system\tcl84t.pdb added.   Debugging info. v8.4.15

5) File system\php4ts.dll changed. Upgraded php to version 4.4.7.

6) File system\dbghelp.dll added.  Support for minidumps.  v6.8.4.0.
   Redistributable from the MS Debugging Tools for Windows package.

7) File system\symsrv.dll added.   Support for remote symbol lookups. v6.8.4.0
   Redistributable from the MS Debugging Tools for Windows package.

8) File system\makecert.exe added. Generates self-signed SSL certificates.
   v5.131.3790.0.  It appears that unlike CertMgr.exe which requires you to
   download the entire platform SDK (300+ MB for the older XP release or
   1GB+ for Vista) you can distribute just makecert.exe via the old original
   XP redist license or via the VS2005 license.  See "site removecert"
   below for ways to run a GUI to manipulate installed certs.

9) File system\ioFTPD.ini changed. Lots of changes detailed below, and I
   re-ordered the sections just a little to put the "must-change" stuff right
   at the top.

10) File system\ServiceInstaller.exe changed.

11) Directory source\include\* changed.

12) File ioKnock\ioKnock.exe added. Version 1.0.0.0.

13) File ioFTPD-Start.exe changed. Version 1.1.0.0.

14) File README changed.


*** Functionality changes

15) Zero byte files are no longer created if no data connection was
    established.  This should prevent most of those pesky 0 byte files.

16) Existing zero byte files that are the object of a STOR command (upload)
    will no longer apply the Overwrite rule but instead use the Resume rule.
    This will allow less permissive settings in the .ini file for average
    users.  The OnUpload event instead of OnResume will still be called though.
    Formally requesting a resume from 0 and then a STOR will continue to use
    the resume rule and the OnResume event.  I haven't seen any FTP clients
    do this though.

17) Resuming a file you do not have permission to write to in the VFS (i.e.
    somebody else's file) now returns an immediate file permission error.
    Previously it would signal all is OK, wait for the data connection to be
    established then close the connection with a transfer complete response
     without actually doing anything.

18) Resuming a file or overwriting a zero byte file now updates the owner
    of the file to the resuming user.

19) A successful non-anonymous login now resets the auto-ban connection
    count for the associated IP.  This should hopefully reduce the chance
    a user with 8-10 connections which all get dropped at the same time
    from getting auto-banned when reconnecting.

20) Only 1 "Rejected auto-banned IP..." message per minute per host will be
    added to the error logfile.  Previously each rejection was logged.

21) Only 1 "Rejected unmatched client..." message per minute per host will
    be added to the error logfile.  Previous each rejection was logged.

22) Only 1 "Rejected client from..." message per minute per host will
    be added to the error logfile.  Previous each rejection was logged.

    NOTE: These 3 error messages use separate timers.  This is particularly
    important when a user with an unmatched host mask tries to login several
    times and gets himself auto-banned.  That message would be suppressed if
    they shared one timer...

23) Changed the "site change" commands to enable the modification of any user
    account if done from a Master user.  Previously Master accounts that were
    granted specific group admin rights would be unable to modify accounts
    outside of the indicated groups.

24) Fixed the MDTM command.  Previously it wouldn't report or modify the
    timestamp.

25) Changed the way the Bind= option in ioFTPD.ini is handled.  This is the
    local IP address to actually create the socket and listen for connections
    on.  Previously if you did not define this it would use the Host= field.
    Since so many sites are behind NAT firewalls failing to specify 0.0.0.0
    for Bind= when specifying an external ip or dynamic dns name could cause
    the server to fail to start properly or result in unroutable sockets.
    Now if you do not specify a Bind= option it will default to all local
    interfaces (which is good) and you need only set this option in those
    specific situations where you must force a particular interface to be
    used.  Changed ioFTPD.ini description of how to setup Host/Bind options.

26) Modified behavior for finding a SSL certificate to use.  If the service
    specifies a Certificate_Name (the normal place to define it) then that is
    tried first.  If not valid or not specified then the Host= value for the
    device associated with the service is tried provided it is different than
    the generic ANYHOST value of 0.0.0.0.  Finally as a fallback the default
    name of "ioFTPD" will be used.

27) Errors in loading the TLS/SSL library (the SSPI stuff), opening the
    certificate store, or creating the server/client credentials (i.e.
    you didn't specify a valid certificate name) are now logged to the
    error logfile.  This should hopefully provide more information for
    users having issues setting up TLS/SSL.

28) A directory is considered "empty" if it consists only of files starting
    with ".ioFTPD", thumbs.db, and/or desktop.ini.  Previously the two hidden
    system files would prevent a directory that looked empty from being
    deleted.

29) An "empty" directory that is marked read-only can now be deleted.  The
    reason this is necessary is because customized folders (such as from
    "Customize this Folder...") specifically enable the read-only
    bit for the directory and fail to clear it even if customizations are
    removed.  This is tricky to notice since ALL directories claim they are
    read-only when you examine their properties in explorer.

30) Crashes will no longer result in the c:\ioFTPD.crash.log file being
    created.

31) The error log entry for hosts not matching the user mask changed from
    "Host 'IDENT@HOST-ELSE-IP" did not match any of user 'USER' allowed hosts."
    to
    "Host 'IDENT@IP' (HOST) did not match any of user 'USER' allowed hosts."


*** New Features

32) New ioFTPD.ini option (Max_Resolver_Threads under [Network]).  Added an
    option to change the maximum number of worker threads that may be used to
    resolve hostnames.  Large heavily used sites may wish to raise this.

33) New ioFTPD.ini option (Immune_Hosts under [Network]).  List of space
    separated wildcard IPs/hosts that should be immune from auto-banning.

34) New ioFTPD.ini option (Immune_Users under [Network]).  A full ioFTPD
    user permission specifier to indicate users whose IP/host masks should be
    immune from auto-banning.  Be VERY careful here since just one immune
    user with *@* now or in the future effectively turns off everything.  I
    don't suggest using this option.

35) New site command (site bans immune) displays the immune IP / host list
    and indicate the source [.ini->immune_hosts, username->immune_users match].

36) New ioFTPD.ini option (Secure_Ip_# under [Network]).  A series of optional
    rules used to enforce what user IP/Host masks must look like depending on
    who is changing the user account.  Similar to glftpd's secure_ip option,
    but more powerful.  See the .ini file for full documentation.

37) If site addip rejects an IP/Host mask because Secure_Ip is enabled and
    no rule matches then a human readable form of the rules is displayed.

38) Site addip will now remove and/or reject redundant entries.  For example
    if a user already has *@* it makes no sense to allow the addition of 
    *@1.2.3.4 (new).  Similarly if you add *@* all other existing entries
    will be removed automatically (existing behavior).  Attempts to add an
    exactly matching entry will no longer result in remove/add events in the
    admin log since nothing actually changed.

39) New site command (site MakeCert).  First determine the preferred name for
    the certificate by looking for a Certificate_Name= for the service, or
    the Host= value for the device specified for the service provided it isn't
    0.0.0.0, or finally the default value of ioFTPD.  Then create and install
    in the CurrentUser My/Personal store a new certificate if one doesn't
    already exist.  Requires the file system/makecert.exe and the M flag.

[Yil]

40) New site command (site RemoveCert name).  If no SSL connections are in
    use but a SSL cert is loaded then this command will attempt to unload the
    cert and remove it from the CurrentUser My/Personal store.  Primarily
    useful for removing certificates stored under the Local System account
    when running ioFTPD as a service.  Requires the M flag and the cert name
    must be one of the checked names used by makecert.  Thus you can't
    delete arbitrary certificates.

    NOTE: To view installed certificates for regular users you can use the
    certificate snap-in for the Microsoft Management Console as this should be
    available on W2k+.  (start->run, "MMC", File->Add/Remove Snap-In, Add,
    Certificates, Add, My user account, Finish, Close, OK.  As a shortcut you
    can try from start->run prompt "certmgr.msc" which will allow you to
    manage the certs for the current user only with fewer steps. 

    NOTE: I know of no way to do anything related to the LocalSystem account
    using the default store using any GUI tool.  Thus if you have installed
    ioFTPD as a service and are running it under the default LocalSystem
    account site makecert/removecert are your friends.  For security purposes
    you probably should be using a limited rights account anyway though...

41) New ioFTPD.ini option (Obscure_IP under [Network]).  This allows you to
    specify how many elements of the dotted IP address should be obscured in
    all logfiles.  Thus instead of 1.2.3.4 it will show up as 1.2.3.* or
    1.2.*.*, etc.  A setting of >=4 will result in "-hidden-" being shown.

42) New ioFTPD.ini option (Obscure_Host under [Network]).  This allows you to
    specify how many elements of a printed hostname should be obscured in
    all logfiles.  Thus instead of foo.bar.com it will show up as *.bar.com
    or *.com.  If the entire hostname is obscured the text "-hidden-" will
    be shown instead.

    NOTE: The combination of Obscure_IP and Obscure_Host should remove the
    need for any locally run BNCs.  However their use obscures even items
    like auto-ban messages and user hostmask mismatch messages which can
    be annoying at times.

43) New ioFTPD.ini option (Dynamic_DNS_Lookup under [Network]).  If a user
    hostmask begins with a colon ":" and is a hostname without any wildcards
    then during the login event you can force a DNS lookup of this hostname
    to see if matches the user's IP.  This should add support for people who
    use dynamic DNS services like no-ip.com.

44) New ioFTPD.ini option (Knock_# under [Network]).  When running a server
    with Reject_Unknown_Ips there is no way to even get to a login prompt
    if your IP has changed.  This can now be a common problem if using just
    a dynamic IP address hostmask.  The solution is a very simple knock-
    knock system which will add the knocking IP to a temporary list so
    you can connect.  Knocking essentially means connecting via TCP to
    between 1 and 5 ports in a short amount of time.  This can easily be
    done in most FTP programs by just setting up fake ftp servers on the
    knock ports and trying to connecting in order, or using the ioKnock
    GUI on windows machines.

45) New executable (ioKnock.exe).  This creates a very simple frontend
    that allows you to store knock sequences for various sites and to
    easily execute them.  With no arguments it pops up a GUI, or with
    1 command line argument it will attempt to locate a saved site by
    that name and execute the knock.

46) Support for enhanced crash reporting added.  Instead of writing to the
    c:\ioFTPD.crash.log file with just the exception ID and crash address
    the server will now create 2 or 3 new files...

47) New output file (CRASH-Log.txt) generated on program crashes.  Contains
    a human readable reason for the crash, useful information like the
    operating system version and processor information, address ranges of
    all the loaded modules for the program, as well as stack backtrace info.

    It looks something like this:

Fri Sep 21 02:32:12 2007 - ioFTPD v6.3.0
Unhandled exception: Access Violation (0xC0000005)
Address: 0x00436A4D [attempting to write data to 0x00000000]
PID=1988, PATH=c:\Projects\ioFTPD6\ioFTPD\system\ioFTPD-debug.exe

System information:
Processor #0 Name: AMD Athlon(tm) 64 Processor 3000+
Processor #0 Identifier: x86 Family 15 Model 12 Stepping 0
OS: Windows 0.0 (build 0)
Registry: Microsoft Windows XP
Decoded: XP - Service Pack 2
Page size: 4096

Modules:
--------
[00400000 - 004d8000]: c:\Projects\ioFTPD6\ioFTPD\system\ioFTPD.exe (v6.3.0.0)
Rest of Modules...

Threads:
--------

ID: 2608  [00130000-0012fe94]
  # 1: 7C90EB94 -> [ntdll + DB94] _KiFastSystemCallRet@0() + 0x0
  # 2: 7E4191BE -> [USER32 + 81BE] _NtUserGetMessage@16() + 0xC
  # 3: 00416B50 -> [ioFTPD + 15B50] ProcessMessages() + 0x40
                    [c:\projects\ioftpd6\6.3.0\src\internalmessagehandler.c, line 106]
  # 4: 0041A2B0 -> [ioFTPD + 192B0] CommonMain() + 0x30
                    [c:\projects\ioftpd6\6.3.0\src\main.c, line 250]
  # 5: 0041A6F3 -> [ioFTPD + 196F3] WinMain() + 0x273
                    [c:\projects\ioftpd6\6.3.0\src\main.c, line 390]
  # 6: 004431E9 -> [ioFTPD + 421E9] __tmainCRTStartup() + 0x177
                    [f:\sp\vctools\crt_bld\self_x86\crt\src\crt0.c, line 324]
  # 7: 7C816FD7 -> [kernel32 + 15FD7] _BaseProcessStart@4() + 0x23
Rest of Threads...


48) New output file (MINIDUMP-date.time-pid.dmp).  Contains a near perfect
    image of the application at the time of the crash.  Size is roughly 4MB
    using the provided dbghelp.dll for a simple server.  Larger sites with
    a large directory cache size will obviously be bigger.

49) New output file (TINYDUMP-date.time-pid.dmp).  Contains just basic
    information, the thread stacks, and variables directly referenced by
    things on the stacks.  Size is roughly 200k using the provided dbghelp.dll.

50) New ioFTPD.ini option (Crash_Dir under [Locations]).  Allows you to
    specify the directory to create CRASH-Log.txt, MINIDUMP* and TINYDUMP*
    files in.  Default is directory where ioFTPD.exe is located.

51) New ioFTPD.ini option (TinyDump_Only under [Locations]).  Allows you to
    disable the generation of the MINIDUMP* files.  This is disabled by
    default since many problems can only be solved with this larger memory
    image.

[Yil]

52) New ioFTPD.ini option (Symbol_Path under [Location]).  This is very
    similiar to the _NT_SYMBOL_PATH environmental variable and thus useful
    when running ioFTPD as a system service.

    When writing stack backtrace information to CRASH-Log.txt it's nice to
    see correct function names instead of just addresses.  This is only
    required for the text in CRASH-Log.txt to display accurately.  If you
    submit Tiny/Minidumps for debugging they can retrieve the correct
    information regardless!

    With no help the stack trace for the original thread in ioFTPD looks like:

    ID: 3708  [00130000-0012fe94]
     # 1: 7C90EB94 -> [ntdll + DB94] ? KiFastSystemCallRet() + 0x0
     # 2: 00416B50 -> [ioFTPD + 15B50]
     # 3: 00443219 -> [ioFTPD + 42219]
     # 4: 7C816FD7 -> [kernel32 + 15FD7] ? RegisterWaitForInputIdle() + 0x49

    If the ioFTPD.pdb, tcl84t.pdb, etc files are in the directory where
    ioFTPD.exe is run from then function names and line numbers can be
    displayed for those files. Resulting in:

    ID: 1212  [00130000-0012fe94]
     # 1: 7C90EB94 -> [ntdll + DB94] ? KiFastSystemCallRet() + 0x0
     # 2: 00416B50 -> [ioFTPD + 15B50] ProcessMessages() + 0x40
                      [c:\projects\ioftpd6\6.3.0\src\internalmessagehandler.c, line 106]
     # 3: 0041A2E0 -> [ioFTPD + 192E0] CommonMain() + 0x30
                      [c:\projects\ioftpd6\6.3.0\src\main.c, line 250]
     # 4: 0041A723 -> [ioFTPD + 19723] WinMain() + 0x273
                      [c:\projects\ioftpd6\6.3.0\src\main.c, line 390]
     # 5: 00443219 -> [ioFTPD + 42219] __tmainCRTStartup() + 0x177
                      [f:\sp\vctools\crt_bld\self_x86\crt\src\crt0.c, line 324]
     # 6: 7C816FD7 -> [kernel32 + 15FD7] ? RegisterWaitForInputIdle() + 0x49

     The ? indicates dll's that do not have loaded debugging info and thus the
     name printed is the first previous exported symbol found which is often
     correct for the entry point into a library but usually wrong after that.

     With system symbol searching enabled the same stack looks like:
     ID: 364   [00130000-0012fe94]
      # 1: 7C90EB94 -> [ntdll + DB94] _KiFastSystemCallRet@0() + 0x0
      # 2: 7E4191BE -> [USER32 + 81BE] _NtUserGetMessage@16() + 0xC
      # 3: 00416B50 -> [ioFTPD + 15B50] ProcessMessages() + 0x40
                       [c:\projects\ioftpd6\6.3.0\src\internalmessagehandler.c, line 106]
      # 4: 0041A2E0 -> [ioFTPD + 192E0] CommonMain() + 0x30
                       [c:\projects\ioftpd6\6.3.0\src\main.c, line 250]
      # 5: 0041A723 -> [ioFTPD + 19723] WinMain() + 0x273
                       [c:\projects\ioftpd6\6.3.0\src\main.c, line 390]
      # 6: 00443219 -> [ioFTPD + 42219] __tmainCRTStartup() + 0x177
                       [f:\sp\vctools\crt_bld\self_x86\crt\src\crt0.c, line 324]
      # 7: 7C816FD7 -> [kernel32 + 15FD7] _BaseProcessStart@4() + 0x23

    Notice that even the number of stack frames is incorrect without access
    to the .pdb file for optimized code.

    See the .ini file for how to setup a local symbol cache, enable access
    to MS's downloadable symbol store temporarily so you can download the
    appropriate symbol files once, and then set it to only use the local
    files from then on.

53) New site command (site LoadSymbols).  Very similar to "site CrashNow"
    except it will force the resolving of all modules, not just those
    currently referenced on the stack, and display additional information
    such as the path to the located PDB for each module used for resolving
    in the generated CRASH-log.txt file.  Limited to master (M) accounts only.

54) Register the Allocate, ReAllocate, and Free procedures to the module
    system.  Should fix problems with user modules.  Not sure if it will
    work if _DEBUG_MEM is enabled when compiling ioFTPD (it's not by default).

55) ioFTPD-Start now defaults to a 10 (was 5) second delay before starting
    ioGUI, and also takes an optional command line argument which is the
    number of seconds to delay.


*** Internal changes

56) Moved version information lookup from the executable's resources from the
    site ioversion command to the main startup function to support printing
    version information in crash logs and eliminate redundant lookups.

57) Changed the logfile write of denied connections for auto-banned IP
    addresses to be done outside of the critical section lock.

58) Fixed a bug during initialization which resulted in a call to closesocket
    with an invalid socket handle.

59) Fixed a bug in the shared memory allocator of debug builds that could
    result in an incorrect share count which could eventually lead to
    improperly freed memory.

60) Removed old #ifdefs and defines for Registered versions only code.

61) About 100 other little things I forgot to write down...

[Yil]

Quick note: ioKnock.exe seems to have a runtime dependency on a VS2005 library so some people may have trouble running it. Really quick googling shows it's a common problem and some people report having .NET 2.0 installed fixes things.

I solved this for ioFTPD a long time ago, but because the GUI is using MFC I'm not sure how to do this yet since it complains when trying to link to a static CRT...

[rolan]

nice work, thx, will try it...

[fr0st]

you rule yil!

[zkx]

nice, i'm sure that debugging stuff will come in handy... and the usual "Source?"
and that ip/host obscure stuff looks nice

nice work

[Yil]

Source:
ioFTPD-v6.3.0-src.zip

[isteana]

the killing scripts times are still 2minute?
or could you make new option as define killing scripts time
cuase some scripts need very long time for make finish
ex) site rebuild on nxtools or site createlog on ioA
that makes log of all dir/files in vfs for site search/dupe
so bigger dir/files needs more time to make log
then usually if over 5000 dirs, script going dead(server time out)cuase over 2 minute when creating dupe log

[Yil]

isteana: I'm not positive how ioFTPD handles timeouts. A really quick look at the EXEC module seems to indicate a script can run as long as it wants provided it outputs something at least every 2 minutes and the client is reading the output...

Do the scripts in question in fact print something while working? If not, then I suggest they should be modified to do so and see what happens. Removing the timeout just seems like a really bad idea, although I'm open to making it an option.

On the other hand clients like FlashFXP often terminate connections when the server appears unresponsive which may be as short as 1-2 minutes. I had to add status output to long running internal commands like file moving, etc to make the clients happy...

As an aside, it looks like the EXEC module doesn't terminate the started child processes under any condition but does close the stdin handle. This probably means the command will actually finish even if you are disconnected...

[isteana]

Yil, anyway make a that option plz if you could

the daemon dropping me if over 2 minute then script does not works anymore its indeed if even using any clients any options


then another old issue below

i really hope 6.3.x will support unicode option(raw incoding) for TCL

you can refer this page below for directly return to External (this page not for only english..but you can watch that code)

http://tcltk.co.kr/moniwiki/wiki.php...CE%C4%DA%B5%F9

then could you also make that raw incoding option?

[Flow]

You Rock Yil! thanks for the lovely new toy and fixes.

[Linkster]

one word....IMPRESSIVE. So nice to see such amazing headway being made.

[noelek]

hi, in Error.log I have sometimes bad reverse name

11-03-2007 11:53:57 Rejected unmatched client 88.xx.xx.xx (X˙
).
11-03-2007 11:55:57 Rejected unmatched client 88.xx.xx.xx ().
11-04-2007 13:59:34 Rejected unmatched client 84.xx.xx.xx (X˙
).

thanks for your good work