1
0
Fork 0
mirror of https://github.com/deepfakes/faceswap synced 2025-06-09 04:36:50 -04:00
faceswap/tools/sort/cli.py
torzdf d8557c1970
Faceswap 2.0 (#1045)
* Core Updates
    - Remove lib.utils.keras_backend_quiet and replace with get_backend() where relevant
    - Document lib.gpu_stats and lib.sys_info
    - Remove call to GPUStats.is_plaidml from convert and replace with get_backend()
    - lib.gui.menu - typofix

* Update Dependencies
Bump Tensorflow Version Check

* Port extraction to tf2

* Add custom import finder for loading Keras or tf.keras depending on backend

* Add `tensorflow` to KerasFinder search path

* Basic TF2 training running

* model.initializers - docstring fix

* Fix and pass tests for tf2

* Replace Keras backend tests with faceswap backend tests

* Initial optimizers update

* Monkey patch tf.keras optimizer

* Remove custom Adam Optimizers and Memory Saving Gradients

* Remove multi-gpu option. Add Distribution to cli

* plugins.train.model._base: Add Mirror, Central and Default distribution strategies

* Update tensorboard kwargs for tf2

* Penalized Loss - Fix for TF2 and AMD

* Fix syntax for tf2.1

* requirements typo fix

* Explicit None for clipnorm if using a distribution strategy

* Fix penalized loss for distribution strategies

* Update Dlight

* typo fix

* Pin to TF2.2

* setup.py - Install tensorflow from pip if not available in Conda

* Add reduction options and set default for mirrored distribution strategy

* Explicitly use default strategy rather than nullcontext

* lib.model.backup_restore documentation

* Remove mirrored strategy reduction method and default based on OS

* Initial restructure - training

* Remove PingPong
Start model.base refactor

* Model saving and resuming enabled

* More tidying up of model.base

* Enable backup and snapshotting

* Re-enable state file
Remove loss names from state file
Fix print loss function
Set snapshot iterations correctly

* Revert original model to Keras Model structure rather than custom layer
Output full model and sub model summary
Change NNBlocks to callables rather than custom keras layers

* Apply custom Conv2D layer

* Finalize NNBlock restructure
Update Dfaker blocks

* Fix reloading model under a different distribution strategy

* Pass command line arguments through to trainer

* Remove training_opts from model and reference params directly

* Tidy up model __init__

* Re-enable tensorboard logging
Suppress "Model Not Compiled" warning

* Fix timelapse

* lib.model.nnblocks - Bugfix residual block
Port dfaker
bugfix original

* dfl-h128 ported

* DFL SAE ported

* IAE Ported

* dlight ported

* port lightweight

* realface ported

* unbalanced ported

* villain ported

* lib.cli.args - Update Batchsize + move allow_growth to config

* Remove output shape definition
Get image sizes per side rather than globally

* Strip mask input from encoder

* Fix learn mask and output learned mask to preview

* Trigger Allow Growth prior to setting strategy

* Fix GUI Graphing

* GUI - Display batchsize correctly + fix training graphs

* Fix penalized loss

* Enable mixed precision training

* Update analysis displayed batch to match input

* Penalized Loss - Multi-GPU Fix

* Fix all losses for TF2

* Fix Reflect Padding

* Allow different input size for each side of the model

* Fix conv-aware initialization on reload

* Switch allow_growth order

* Move mixed_precision to cli

* Remove distrubution strategies

* Compile penalized loss sub-function into LossContainer

* Bump default save interval to 250
Generate preview on first iteration but don't save
Fix iterations to start at 1 instead of 0
Remove training deprecation warnings
Bump some scripts.train loglevels

* Add ability to refresh preview on demand on pop-up window

* Enable refresh of training preview from GUI

* Fix Convert
Debug logging in Initializers

* Fix Preview Tool

* Update Legacy TF1 weights to TF2
Catch stats error on loading stats with missing logs

* lib.gui.popup_configure - Make more responsive + document

* Multiple Outputs supported in trainer
Original Model - Mask output bugfix

* Make universal inference model for convert
Remove scaling from penalized mask loss (now handled at input to y_true)

* Fix inference model to work properly with all models

* Fix multi-scale output for convert

* Fix clipnorm issue with distribution strategies
Edit error message on OOM

* Update plaidml losses

* Add missing file

* Disable gmsd loss for plaidnl

* PlaidML - Basic training working

* clipnorm rewriting for mixed-precision

* Inference model creation bugfixes

* Remove debug code

* Bugfix: Default clipnorm to 1.0

* Remove all mask inputs from training code

* Remove mask inputs from convert

* GUI - Analysis Tab - Docstrings

* Fix rate in totals row

* lib.gui - Only update display pages if they have focus

* Save the model on first iteration

* plaidml - Fix SSIM loss with penalized loss

* tools.alignments - Remove manual and fix jobs

* GUI - Remove case formatting on help text

* gui MultiSelect custom widget - Set default values on init

* vgg_face2 - Move to plugins.extract.recognition and use plugins._base base class
cli - Add global GPU Exclude Option
tools.sort - Use global GPU Exlude option for backend
lib.model.session - Exclude all GPUs when running in CPU mode
lib.cli.launcher - Set backend to CPU mode when all GPUs excluded

* Cascade excluded devices to GPU Stats

* Explicit GPU selection for Train and Convert

* Reduce Tensorflow Min GPU Multiprocessor Count to 4

* remove compat.v1 code from extract

* Force TF to skip mixed precision compatibility check if GPUs have been filtered

* Add notes to config for non-working AMD losses

* Rasie error if forcing extract to CPU mode

* Fix loading of legace dfl-sae weights + dfl-sae typo fix

* Remove unused requirements
Update sphinx requirements
Fix broken rst file locations

* docs: lib.gui.display

* clipnorm amd condition check

* documentation - gui.display_analysis

* Documentation - gui.popup_configure

* Documentation - lib.logger

* Documentation - lib.model.initializers

* Documentation - lib.model.layers

* Documentation - lib.model.losses

* Documentation - lib.model.nn_blocks

* Documetation - lib.model.normalization

* Documentation - lib.model.session

* Documentation - lib.plaidml_stats

* Documentation: lib.training_data

* Documentation: lib.utils

* Documentation: plugins.train.model._base

* GUI Stats: prevent stats from using GPU

* Documentation - Original Model

* Documentation: plugins.model.trainer._base

* linting

* unit tests: initializers + losses

* unit tests: nn_blocks

* bugfix - Exclude gpu devices in train, not include

* Enable Exclude-Gpus in Extract

* Enable exclude gpus in tools

* Disallow multiple plugin types in a single model folder

* Automatically add exclude_gpus argument in for cpu backends

* Cpu backend fixes

* Relax optimizer test threshold

* Default Train settings - Set mask to Extended

* Update Extractor cli help text
Update to Python 3.8

* Fix FAN to run on CPU

* lib.plaidml_tools - typofix

* Linux installer - check for curl

* linux installer - typo fix
2020-08-12 10:36:41 +01:00

188 lines
11 KiB
Python

#!/usr/bin/env python3
""" Command Line Arguments for tools """
from lib.cli.args import FaceSwapArgs
from lib.cli.actions import DirFullPaths, SaveFileFullPaths, Radio, Slider
_HELPTEXT = "This command lets you sort images using various methods."
class SortArgs(FaceSwapArgs):
""" Class to parse the command line arguments for sort tool """
@staticmethod
def get_info():
""" Return command information """
return "Sort faces using a number of different techniques"
@staticmethod
def get_argument_list():
""" Put the arguments in a list so that they are accessible from both argparse and gui """
argument_list = list()
argument_list.append({"opts": ('-i', '--input'),
"action": DirFullPaths,
"dest": "input_dir",
"group": "data",
"help": "Input directory of aligned faces.",
"required": True})
argument_list.append({"opts": ('-o', '--output'),
"action": DirFullPaths,
"dest": "output_dir",
"group": "data",
"help": "Output directory for sorted aligned "
"faces."})
argument_list.append({"opts": ('-s', '--sort-by'),
"action": Radio,
"type": str,
"choices": ("blur", "face", "face-cnn", "face-cnn-dissim",
"face-yaw", "hist", "hist-dissim", "color-gray",
"color-luma", "color-green", "color-orange"),
"dest": 'sort_method',
"group": "sort settings",
"default": "face",
"help": "R|Sort by method. Choose how images are sorted. "
"\nL|'blur': Sort faces by blurriness."
"\nL|'face': Use VGG Face to sort by face similarity. This "
"uses a pairwise clustering algorithm to check the "
"distances between 4096 features on every face in your set "
"and order them appropriately. WARNING: On very large "
"datasets it is possible to run out of memory performing "
"this calculation."
"\nL|'face-cnn': Sort faces by their landmarks. You can "
"adjust the threshold with the '-t' (--ref_threshold) "
"option."
"\nL|'face-cnn-dissim': Like 'face-cnn' but sorts by "
"dissimilarity."
"\nL|'face-yaw': Sort faces by Yaw (rotation left to right)."
"\nL|'hist': Sort faces by their color histogram. You can "
"adjust the threshold with the '-t' (--ref_threshold) "
"option."
"\nL|'hist-dissim': Like 'hist' but sorts by dissimilarity."
"\nL|'color-gray': Sort images by the average intensity of "
"the converted grayscale color channel."
"\nL|'color-luma': Sort images by the average intensity of "
"the converted Y color channel. Bright lighting and "
"oversaturated images will be ranked first."
"\nL|'color-green': Sort images by the average intensity of "
"the converted Cg color channel. Green images will be "
"ranked first and red images will be last."
"\nL|'color-orange': Sort images by the average intensity "
"of the converted Co color channel. Orange images will be "
"ranked first and blue images will be last."
"\nDefault: hist"})
argument_list.append({"opts": ('-k', '--keep'),
"action": 'store_true',
"dest": 'keep_original',
"default": False,
"group": "output",
"help": "Keeps the original files in the input "
"directory. Be careful when using this "
"with rename grouping and no specified "
"output directory as this would keep "
"the original and renamed files in the "
"same directory."})
argument_list.append({"opts": ('-t', '--ref_threshold'),
"action": Slider,
"min_max": (-1.0, 10.0),
"rounding": 2,
"type": float,
"dest": 'min_threshold',
"group": "sort settings",
"default": -1.0,
"help": "Float value. "
"Minimum threshold to use for grouping comparison with "
"'face-cnn' and 'hist' methods. The lower the value the "
"more discriminating the grouping is. Leaving -1.0 will "
"allow the program set the default value automatically. "
"For face-cnn 7.2 should be enough, with 4 being very "
"discriminating. For hist 0.3 should be enough, with 0.2 "
"being very discriminating. Be careful setting a value "
"that's too low in a directory with many images, as this "
"could result in a lot of directories being created. "
"Defaults: face-cnn 7.2, hist 0.3"})
argument_list.append({"opts": ('-fp', '--final-process'),
"action": Radio,
"type": str,
"choices": ("folders", "rename"),
"dest": 'final_process',
"default": "rename",
"group": "output",
"help": "R|Default: rename."
"\nL|'folders': files are sorted using "
"the -s/--sort-by method, then they "
"are organized into folders using "
"the -g/--group-by grouping method."
"\nL|'rename': files are sorted using "
"the -s/--sort-by then they are "
"renamed."})
argument_list.append({"opts": ('-g', '--group-by'),
"action": Radio,
"type": str,
"choices": ("blur", "face-cnn", "face-yaw", "hist"),
"dest": 'group_method',
"group": "output",
"default": "hist",
"help": "Group by method. "
"When -fp/--final-processing by "
"folders choose the how the images are "
"grouped after sorting. "
"Default: hist"})
argument_list.append({"opts": ('-b', '--bins'),
"action": Slider,
"min_max": (1, 100),
"rounding": 1,
"type": int,
"dest": 'num_bins',
"group": "output",
"default": 5,
"help": "Integer value. "
"Number of folders that will be used "
"to group by blur and face-yaw. "
"For blur folder 0 will be the least "
"blurry, while the last folder will be "
"the blurriest. "
"For face-yaw the number of bins is by "
"how much 180 degrees is divided. So "
"if you use 18, then each folder will "
"be a 10 degree increment. Folder 0 "
"will contain faces looking the most "
"to the left whereas the last folder "
"will contain the faces looking the "
"most to the right. "
"If the number of images doesn't "
"divide evenly into the number of "
"bins, the remaining images get put in "
"the last bin."
"Default value: 5"})
argument_list.append({"opts": ('-l', '--log-changes'),
"action": 'store_true',
"group": "settings",
"default": False,
"help": "Logs file renaming changes if "
"grouping by renaming, or it logs the "
"file copying/movement if grouping by "
"folders. If no log file is specified "
"with '--log-file', then a "
"'sort_log.json' file will be created "
"in the input directory."})
argument_list.append({"opts": ('-lf', '--log-file'),
"action": SaveFileFullPaths,
"filetypes": "alignments",
"group": "settings",
"dest": 'log_file_path',
"default": 'sort_log.json',
"help": "Specify a log file to use for saving "
"the renaming or grouping information. "
"If specified extension isn't 'json' "
"or 'yaml', then json will be used as "
"the serializer, with the supplied "
"filename. "
"Default: sort_log.json"})
return argument_list