Cant seem to get the skiplist to work

[brackebuschtino]

Hello, i am trying to get the skiplist to work without luck. I checked the help file and used several regex testers to validate my regexs and found them being valid. But the folders aren't highlighted nor skipped as expected. After reading several threads here in forum i checked what you repeatedly responded:

1. no per site skiplist in use (yet, but planned to have)
2. global skiplist:
   "Enable skiplist" > checked
   "Skip 0 byte files" > not checked
   "Skip emtpy folders" > checked
   "Display skip items in" > red
   "When to skip" > Both
3. Mask > .*[^(substring)]$
4. Compare > Name of Folder

I want to match all folders that DO NOT MATCH the substring. It is planned to extened this substring by other substrings via pipe symbol, which is why i used the grouping. This pattern doesn't work.

This mask doesn't match either.

1. Mask > .*substring$
2. Compare > Name of Folder

I also tried to use 'Path of Folder' for comparison since on the server i am at 3rd or 4th sublevel below root. But this didn't fix the issue.

I don't understand why and feel pretty helpless since i cannot debug the transfer. Where is my mistake?

[bigstar]

FlashFXP uses a tiny subset of the regular expression syntax.

It doesn't support a reverse evaluation, such as matching everything but <pattern>

See FlashFXP Help File - Pattern Matching for more information.

[brackebuschtino]

With the help of that site i created my non-working regexp. I know it is not supporting reverse evaluation which is why i tried to combine the substrings as groups inside the [^<char set>] notation, which means to match if not containing any of the characters in <char set>. So, if this is supported, why don't groups of substrings like [^<(group1|group2|group3)>] work? Matching a single substring and matching a group of substrings works, Somehow it should be possible to negate that expression, shouldn't it?

Anyway, taking your suggestion to use *.*substring* instead of .*substring$ doesn't work either. The folder containing the substring doesn't get highlighted in red - neither when comparing by Name of Folder nor Path of Folder.

Please, can you tell me, what to configure for a server path of /level1/Data_From_(2014)-Substring taking care for the data inside that can have any extension and also futher directories?

[bigstar]

If the pattern you're trying to match ends with "-Substring" then you could do *-substring and compare by any

If that works you can try to refine it by using name of folder instead of any and see if that provides you with satisfactory results.

[brackebuschtino]

That works. Now the final problem is to negate this expression in either way just to drop these.
If this is really impossible. Would you mind to implement an allow list so one can save oneself from writing down hundreds of entries into the skiplist just to enable the filtering of only one match? A allowlist would solve the circumstance of unsupported regexes.

What you say?

[bigstar]

I think it would be more productive to add PCRE (Perl Compatible Regular Expressions) support.

This is something I can implement right now in FlashFXP v5.

My thought is to add support using the following syntax "regex:<pattern>", this way both can co-exist.

Here's a v5 build with PCRE support using the regex:<pattern> syntax as mentioned above.
http://get.flashfxp.com/5.0/FlashFXP50_3724_Setup.exe

Currently regex is limited to the skip list and highlights but I will add support for everywhere else in the next update probably tomorrow or monday.

[brackebuschtino]

PCRE support sounds great. Thanks for providing a beta that fast. Given that positive and negative lookahead is supported then? I installed the linked beta version and added the filter

.*-(?!(pattern1|pattern2))

to match all subdirectories names that do not match any of the grouped patterns, but get no results.

A general thought regarding your solution:

Actually i highly appreciate extended or full RegExp support. But, I suppose, this is the faster way to implement my suggestion regarding an allowlist. However, i'm afraid that extended regex capability will also require more support for non-experienced users that want to have some kind of allow list. From the logical aspect skipping content to me means to skip "everything that DOES NOT MATCH a rule" specified as filter. Whereas an allowlist to me means to allow "everything that DOES MATCH a rule" specified as filters.

From the UX point of view one never knows what kind of content a directory may contain. So, making use of the skiplist requires that a server contain semantically equal content so that defined rules will - once defined - skip content as wished. But since many public servers host content one can never know in advance the better way to filter content is to define all stuff one would like to allow for download. This would dramatically decrease the effort required to define skip rules since one does not need to sit by and watch if some content may slip through since its pattern was forgotten to be added to the skiplist.

Of course i do understand that coding another configuration level globally and per site requires more effort from you, but i'm pretty sure that in the end users - especially non-developers - were much more thankful for this feature than for the PCRE support alone. I think you get what i mean.

[bigstar]

Make sure that the line you add into the skip list or highlights is added as

regex: .*-(?!(pattern1|pattern2))

As I previously indicated regex support in this build only works in the skip list or highlights.

I plan to expand that to the rest of the program today.

What might be more suited for what you desire is to use the Selective Transfer feature.

FlashFXP > Tools > Selective Transfer > Edit

This allows you to create multiple rule sets, a rule set is a individual set of rules that determine whether or not a file is skipped or transferred. Rule sets are completely independent of the skip list.

You can quickly switch between active rule sets via the dropdown arrow to the left of the "transfer queue" toolbar button.

You can also bind selective transfer rule sets to individual queue items or scheduled tasks.

The important thing to remember about selective transfer rules is that by default the default rule set is always used unless you manually switch to another rule set or change the rule set for the queue items.

The next build will allow regex in the selective transfer rules.

[brackebuschtino]

Originally Posted by bigstar

Make sure that the line you add into the skip list or highlights is added as

regex: .*-(?!(pattern1|pattern2))

I added this expression one including

regex:

and another time without it and both patterns do not make folders appear highlighted in red. I dunno if i do something wrong or if the expression is not correctly processed. I use the exact same settings as posted in my opening post but only with a different regex and nothing is highlighted.

The only one time when a folder was highlighted was when i applied your previous suggestion

If the pattern you're trying to match ends with "-Substring" then you could do *-substring and compare by any

which matched the wrong folder.

Regardint the "Selected transfer":

I tried to manage my need via this feature longer time ago but dropped it since i couldnt figure out how the rules must be expressed (language/pattern/etc.). I also checked the help file but couldnt get any results. Dunno remember anymore if it was for the same reason (no pattern negation).

Can you provide me with an example for my case, please?

[bigstar]

Pattern:

regex: .*-(?!(one|two))

Set the Scope to "Name of folder" to skip the folder, or to "any" to skip the folder and all content within it.

and the result would be
dir-one
dir-two
dir-three
dir-four
something

If its not working like that please give me a little more time to finish these changes and I'll link you an update.

You'll be able to use the pre-existing pattern matching or the new regex pattern matching anywhere within FlashFXP that uses pattern matching.

[brackebuschtino]

Thanks for the response. It is not (yet) working. I will wait now for your next release. Thanks for your effort!

[bigstar]

Here's the update
http://get.flashfxp.com/5.0/FlashFXP50_3725_Setup.exe

I've made a small change to the syntax prefix

rx .*(txt|log)$

notice instead of regex:<space> the prefix has changed to just rx<space>

[bigstar]

I discovered a couple minor bugs.
http://get.flashfxp.com/5.0/FlashFXP50_3726_Setup.exe

[brackebuschtino]

Thanks for the update. I get at least more matches, but not (yet) the desired behavior. When testing the pattern with an online PCRE regex tester i discovered a potential reason for that.
Given these two folders

/subdir/subdir/.../Author1_-_Title1_(1234)-Publisher1
/subdir/subdir/.../Author2_-_Title2_(5678)-Publisher2

And given this regex

rx .*-(?!publisher1)

i would expect that the second folder would be highlighted in red, but in fact both folders are highlighted because the whole path

/subdir/subdir/.../Author1_-_Title1_(1234)-Publisher1
/subdir/subdir/.../Author2_-_Title2_(1234)-Publisher2

matches this pattern.

The desired behavior is this:

/subdir/subdir/.../Author1_-_Title1_(1234)-Publisher1
/subdir/subdir/.../Author2_-_Title2_(1234)-Publisher2

Also, is there an explanation available explaining how the different regex scopes are evaluated? I mean, i would love to understand how the evaluating routine expects to see the line to be evaluated. Get what i mean? Or differently: Beginning from where up to where is the path that is evaluated for "Name of File" pattern? Same for the other patterns. Understanding this would help to build working patterns. Sometimes is only one missing character or character to much that might prevent a pattern from working. And since there is no way to debug the programm searching for the reason might become a search for the needle in a haystack.

[bigstar]

The compare scope is outlined in the help file, F1 from the skip list tab points you to where its explained.
FlashFXP Help

I think what you need is something like this

((.*)-!?(publisher1$|publisher2$))

I found the following website very good at visually seeing the logic path used for the comparison
https://www.debuggex.com/
And they have a very nice cheatsheet
https://www.debuggex.com/cheatsheet/regex/pcre

For quickly testing regex within FlashFXP I found the Mask Select feature (Ctrl+S) works very well and a big plus is that you can test against the file listing to see exactly what is matched based on the selection.

I had thought about a way to test and evaluate expressions and while I could add a way to do this within the filters dialog it would limit the functionality to this single area, since the goal is to have regex supported everywhere there is no simple way to add a test for each situation. Right now I think using the Mask Select is a good way to evaluate the pattern.

I have discovered a couple more places where the new regex style pattern matching isn't working.
Tools > Server file search > result sub-search
View > Active Edits > search
Options > File Associations > File Patterns

As far as I know regex is working every where else, if not please let me know. I will re-test again once these have been corrected, however this probably wont be until Monday at the earliest.

One thing I forgot to mention is that currently the regular expressions are case-sensitive.

This can be changed using the i flag to ignore case, I am not sure if this should to be changed to use case insensitive matching by default or not. However since the original pattern matching is case insensitive it would make sense that should be as well.