FileZilla FTP-server upgrade to 0.9.6b progress

[gglaze]

hi guys,
i've started the work to upgrade the xbmc version of filezilla to the latest filezilla code from their cvs.

i've decided to take the following steps:
1. re-check and fix the baseline with filezilla v0_8_8
2. upgrade to new baseline with filezilla v0_9_8b
3. evaluate the further updates to latest for potential individual upgrades, as desired.

i've pretty much finished the re-baselining with v0_8_8, although i might like to go in and clean up a few things. see the summary below for the full comparison. there are 4 basic categories of things:

1. files that are heavily modified for xbmc - will keep the xbmc versions of these, with some potential cleanup. in particular, i would like to go through all xbmc-specific sections of code, and insert the if(_xbox) condition around all custom blocks of code that don't already have it.

2. files that are not currently in xbmc - will assume these were removed for some purpose, and are not needed - so leaving them out.

3. files that are in xbmc but not in fzilla - will assume these were added for some purpose, and so are needed - so leaving them in.

4. files that are not changed, or only trivially changed - will replace all trivially changed files currently in xbmc with actual files from filezilla, assuming that it is best to use the filezilla release as the baseline for as much as possible. based on my analysis, this should not affect anything more than a few debug items here and there.

let me know if you guys disagree with my approach or have any questions.

i plean to get the initial analysis for comparison with 0_9_8b done over the next day or so, and then hopefully the actual upgrade done by the end of the week, depending on how many conflicts my analysis uncovers.

once the full upgrade is done, fully functional site commands will of course also be incorporated.




xbmc fzilla baseline check - v0_8_8
===================================

xbmc : previously known modified files:

* source\controlsocket.cpp * * * * this_file, numerous xbox customizations and bug fixes
* source\controlsocket.h * * * * * additional xbox functionality - bsdsfv.h, getcommandfromstring(), msfvfile;
* * * * * * * * * * * * * * * * * *sendcurdir() and senddir() seem to be custom functions, should be in if(_xbox)
* source\options.cpp * * * * * * * custom welcome message for xbmc, also some pragma warnings disabled
* source\permissions.cpp * * * * * this_file, xsetfilecachesize calls, rewrote findnextfile loop in getdirectorylisting method,
* * * * * * * * * * * * * * * * * *bug fixes, includes my single-char-drives bug fixes, need to put this inside if(_xbox),
* * * * * * * * * * * * * * * * * *return empty result if(_xbox) for getshortcuttarget.
* source\server.h * * * * * * * * *made showstatus and onservermessage methods virutal
* source\thread.cpp * * * * * * * *added suspendthread implementation (where is this in fz?)
* source\thread.h * * * * * * * * *made thread.run virtual (if _xbox)
* source\transfersocket.cpp * * * *this_file, nolayers, numerous xbox customizations and bug fixes
* source\transfersocket.h * * * * *some functions removed if nolayers, event handlers public if _xbox instead of protected

more modified files:

* source\version.cpp * * * * * * * include xbmc version info, completely replaces fz file
* source\asyncsocketex.cpp * * * * added lots of xbmc stuff, excluded some functionality for xbmc build
* source\server.cpp * * * * * * * *minor cleanup stuff, cleaner initialization and deletion of some pointers
* source\serverthread.cpp * * * * *remove code that raises thread priority (need to use #if xbmc); also this_file...
* source\stdafx.h * * * * * * * * *change stdstring.h, also change some debug defs
*
* source\misc\markupstl.cpp * * * *small bug fix for deleting array pointer, also #undef this_file... *
* source\settings converter\misc\markupstl.cpp * * * *need same fix as in \misc, also debug_new and this_file (why is this file duped?)
* * * * * - lots of diffs in these 2 files, mostly due to conversion between (char *) and lpctstr
* * * * * - also some _debugx business instead of the normal _debug
*
in fz, not in xbmc: (removed from xbmc version)

* *source\gftp\*
* *source\interface\misc\*
* *source\interface\res\*
* *source\interface\optionspage.cpp
* *source\interface\optionspage.h
* *source\interface\optionspasvpage.cpp
* *source\interface\optionspasvpage.h
* *source\misc\stdstring.h
* *source\settings converter\misc\stdstring.h

in xbmc, not in fz: (added to xbmc version)

* *source\new folder\*
* * * *source\new folder\asyncgsssocketlayer.cpp
* * * *source\new folder\asyncgsssocketlayer.h
* * * *source\new folder\asyncsocketexlayer.cpp
* * * *source\new folder\asyncsocketexlayer.h
* *source\interface\childview.cpp
* *source\interface\childview.h
*

trivial changes (whitespace differences):

* *source\accounts.cpp
* *source\admininterface.cpp
* *source\adminlistensocket.cpp
* *source\adminsocket.cpp
* *source\externalipcheck.cpp
* *source\filelogger.cpp
* *source\mfc64bitfix.cpp
* *source\misc\md5.cpp
* *
trivial changes (negligible debug code differences):

* *source\asyncgsssocketlayer.cpp * * * * * * * * * * * * *#define new debug_new
* *source\asyncsocketexlayer.cpp * * * * * * * * * * * * * #undef this_file...
* *source\listensocket.cpp * * * * * * * * * * * * * * * * #undef this_file...
* *source\service.cpp * * * * * * * * * * * * * * * * * * *#define new debug_new
* *source\speedlimit.cpp * * * * * * * * * * * * * * * * * #undef this_file...
* *
* *source\interface\adminsocket.cpp * * * * * * * * * * * *#define new debug_new
* *source\interface\asyncsocketex.cpp * * * * * * * * * * *#define new debug_new, #undef this_file...
* *source\interface\connectdialog.cpp * * * * * * * * * * *#undef this_file...
* *source\interface\entersomething.cpp * * * * * * * * * * #undef this_file...
* *source\interface\filezilla server.cpp * * * * * * * * * #undef this_file...
* *source\interface\groupsdlg.cpp * * * * * * * * * * * * *#undef this_file...
* *source\interface\groupsdlggeneral.cpp * * * * * * * * * #undef this_file...
* *source\interface\groupsdlgspeedlimit.cpp * * * * * * * *#define new debug_new
* *source\interface\mainfrm.cpp * * * * * * * * * * * * * *#undef this_file...
* *source\interface\newuserdlg.cpp * * * * * * * * * * * * #undef this_file...
* *source\interface\offlineaskdlg.cpp * * * * * * * * * * *#undef this_file...
* *source\interface\options.cpp * * * * * * * * * * * * * *#undef this_file...
* *source\interface\optionsadmininterfacepage.cpp * * * * *#undef this_file...
* *source\interface\optionsdlg.cpp * * * * * * * * * * * * #undef this_file...
* *source\interface\optionsgeneralpage.cpp * * * * * * * * #undef this_file...
* *source\interface\optionsgeneralwelcomemessagepage .cpp * #undef this_file...
* *source\interface\optionsgsspage.cpp * * * * * * * * * * #undef this_file...
* *source\interface\optionsloggingpage.cpp * * * * * * * * #undef this_file...
* *source\interface\optionsmiscpage.cpp * * * * * * * * * *#undef this_file...
* *source\interface\optionssecuritypage.cpp * * * * * * * *#undef this_file...
* *source\interface\optionsspeedlimitpage.cpp * * * * * * *#define new debug_new
* *source\interface\speedlimitruledlg.cpp * * * * * * * * *#undef this_file...
* *source\interface\splitex.cpp * * * * * * * * * * * * * *#undef this_file...
* *source\interface\statusctrl.cpp * * * * * * * * * * * * #undef this_file...
* *source\interface\statusview.cpp * * * * * * * * * * * * #undef this_file...
* *source\interface\usersdlg.cpp * * * * * * * * * * * * * #undef this_file...
* *source\interface\usersdlggeneral.cpp * * * * * * * * * *#undef this_file...
* *source\interface\usersdlgspeedlimit.cpp * * * * * * * * #define new debug_new
* *source\interface\userslistctrl.cpp * * * * * * * * * * *#undef this_file...
* *source\interface\usersview.cpp * * * * * * * * * * * * *#undef this_file...
* *source\interface\version.cpp * * * * * * * * * * * * * *#define new debug_new
* *
* *source\misc\mailmsg.cpp * * * * * * * * * * * * * * * * #define new debug_new
* *source\misc\wheatyexceptionreport.cpp * * * * * * * * * #define new debug_new
* *
* *source\settings converter\options.cpp * * * * * * * * * #define new debug_new, #undef this_file...
* *source\settings converter\permissions.cpp * * * * * * * #define new debug_new, #undef this_file...
* *source\settings converter\settings converter.cpp * * * *#define new debug_new

[spiff]

1. files that are heavily modified for xbmc - will keep the xbmc versions of these, with some potential cleanup. in particular, i would like to go through all xbmc-specific sections of code, and insert the if(_xbox) condition around all custom blocks of code that don't already have it.


i know that i didn't do this when i added the fatx-limit thingy. however, search for servers.ftpautofatx and you should find them easily..

[jmarshall]

great. just a quick point. all that #define stuff should have been taken out by now i think?

also, doesn't look as though those "in xbmc but not in fz" files are used for anything at all (at least they aren't included in the project)

cheers,
jonathan

[gglaze]

jm, which #define stuff are you talking about? the debug_new stuff or the this_file stuff, or something else?

regarding the debug_new stuff - this shows up sometimes in fz, but not always - and has been added in a number of files in xbmc that don't have it in fz. i'm guessing this is for debugging object allocation or something. the point is, these are things xbmc has apparently *added* in some cases to fz files that don't otherwise have it.

regarding the this_file stuff - i have no idea what this is for - but it seems that it has been selectively removed on some files in xbmc, while it is still there in fz.

so in both of these cases, assuming there is no other major change in the file that conflicts with what's in fz, i'm going to just replace whatever is in xbmc with what's currently in fz (as of 0_8_8 for now). i'm hoping this won't impact anything, but let me know if i'm missing something...

so i plan to finish this step and check everything in to cvs tonight, with a new re-baselined version that is fully consistent with 0_8_8.

then the next step will be merging 0_9_8b stuff in. theoretically, once i've finished re-baselining to 0_8_8, all files that were not branched off of the official 0_8_8 version can be directly replaced with 0_9_8b versions of those files. i'll go through everything to be sure. but i think the only actual work involved in upgrading to 0_9_8b at this point should be in that short list of 10 or so files that actually are branched off for "significant" changes in xbmc.

spiff - don't worry - i've done it myself too, and there are plenty of custom sections missing the #ifdef(_xbox) declaration. i'll make sure all this gets cleaned up.

[jmarshall]

sorry for the red herring. reason i brought it up is they #define new debug_new etc. was recently added and then removed again throughout the xbmc source (including filezilla).

cheers,
jonathan

[gglaze]

fyi guys, small change - looks like someone had changed execbuiltin() to return a bool, but i already had a version in local that returns an int, with various codes indicating various types of failures. i actually use this in the ftp site implementation, to display a more intelligent error message. so i'm planning to check this back in with the int return codes.

i've done a quick scan of existing code, and didn't come across anything that was actually looking at the return value, but just wanted to bring this to attention in case anyone has anything that might be dependent on that.

so the return is 0 on success, something else on failure.

[Gamester17]

3. evaluate the further updates to latest for potential individual upgrades, as desiredremember to include jmarshall's corruption bug-fix "sometimes file downloads aborted prematurely leading to incomplete files" which was released in filezilla 0.9.8c ;)

[gglaze]

will do. the weird thing is, there doesn't seem to be a 0_9_8c label on the code - the most recent label i can find is 0_9_8b. i see that they have an official 0_9_8c release, so i guess this must just be from the latest files, but unlabeled.

[Bobbin007]

about the debug_new and this_file stuff. their purpose is the get the line and file of memory allocated with "new", when using crt/mfc mem leak tracking. i have added a define to every stdafx.h we have, that does the same thing. i think they can be ignored or even removed.

greets

bobbin007

[tslayer]

gglaze,

there seems to be a bug introduced with your last update.

site commands seem to be broken.. at least for me. can you please check on it. why are we sending a 200 back to the client for all site commands and then subsequently returning yet another status? i think this is causing issue.

#if defined(_xbox)
case command_site:
{
//get command
cstdstring sitecommand, siteargs;
if (!getcommandfromstring(args, sitecommand, siteargs))
return;


{
cstdstring str;
str.format("200 ftp site command called [command=%s, args=%s]", sitecommand.c_str(), siteargs.c_str());
send(str);
clog::log(lognotice, str);
}


should we be doing that send(str) right away? i don't think the ftp protocol allows for server side to send multiple responses for 1 request.

thanks,

ts

[gglaze]

yes, sorry - that was just debugging stuff - i was afraid i had left something like that in - was planning to test the build out today. will make the change and check it in, in the next few hours when i get home.

[tslayer]

thanks a lot :)

found this using linux.. ncftp or lftp. those programs tend to always send site commands for chmod or utime. and that caused some craziness so i was not able to properly ftp new builds over now using xbmc ftp.

i only debugged this from work this morning so i can't do any tests. i'll test later :)

thanks,

ts

[gglaze]

fyi, this should be checked in and fixed. i'm going to test myself and verify in a few minutes.

[tslayer]

thanks.. but try site help or site xbmc.help.

that is still borked it seems.

but it now properly ignores the site commands it does not understand.

ts

[jmarshall]

maybe the site help commands should be issuing the return code for help (i think atm they just dump it out then return 200)?

not sure - i didn't check it all that much in a number of clients. it did work fine when i added it (in flashfxp at least) but it may well be returning the incorrect codes.

cheers,
jonathan

[dylanbox]

hey...
so how is it going? did you make any progress with this thing?

would be nice to know :)

regards

[dylanbox]

:bump: ???

[gglaze]

sorry for the delay in responding - apparently i'm not subscribed to this topic anymore?

some minor patches for fixed site commands have already been put into cvs. the big filezilla upgrade/merge is in progress, and it's just a matter of me finding enough time to get the last few (big) files done.

stay posted for updates..

[dylanbox]

thank you for the update :)

[Livin]

gglaze,
any chance that we could get a separate filezilla server interface build so we do not need to install the entire server on our windows machines just to get the admin tool?

i cannot figure out how to turn off the 'filezilla server.exe' on my machine... i had to find the service in the registry and remove it or to kill the thread using task manager.

what a pita.

[gglaze]

honestly never tried connecting the admin tool up that way, but i will look into it - all the code is in there, so shouldn't be such a big deal.

[dylanbox]

hi...

i just wanted to ask how things are going. has development stopped on this? i would be very happy if the port came out soon :)

thanks
dylan

[gglaze]

so would i - my dev laptop has been in and out of the shop getting serviced. as soon as i get it back in working shape, i'll be back on this. sorry for the delays.

[HarshReality]

sept to nov.... now there is a stretch....

[DonJ]

if it's not to much trouble maybe u should update to 0.9.11 instead of 0.9.8b.

thx

[gglaze]

will do. if anyone is very familiar with these versions, it would be helpful if you could list out a summary of new features and fixes that we should be looking out for in the new versions - that way we can test and verify that all of the relevant stuff was picked up properly. obviously we can start by looking at the release notes / change logs, but if anyone is particularly knowledgeable, please share.

[monkeyman]

Is there any update on this? It's been a while.

[pike]

way to dig up a buried topic. no there's no progress.