To deploy a pipeline, typically there may be a primary script, or a single point of entry that runs the entire pipeline. For instance, in an information science pipeline, the purpose of entry of the code repository should orchestrate and run the information, feature engineering, modeling, and evaluation pipeline in sequence.
Sometimes, you might have to run various kinds of pipelines or make ad-hoc tweaks to the pipeline.
Tweaks may include omitting certain parts of the code or running the pipeline with different parameters. In data science, there might be a training and scoring pipeline or certain runs that require a full or partial refresh of the information.
The trivial solution might be to create multiple primary scripts. Nonetheless, this may end in code duplication and it is tough to keep up the multiple scripts in the long term — provided that there could be many mixtures of tweaks. A greater solution is to have the primary script accept arguments, in the shape of parameter values or flags, and subsequently run the suitable form of pipeline via the Command Line Interface (CLI).
This text is not going to elaborate on how your primary script decides to make use of the arguments but will introduce alternative ways you may pass in arguments to your primary script — you may consider it as your primary script is now a function that accepts parameters! I can even elaborate on the professionals and cons of every method, and supply code samples of learn how to implement the fundamental to advanced usages.
- Using sysargv: The easiest way
- Using argparse: Essentially the most common way
- Using docopts: An alternate way
- Using Typer: The latest and simplest way
The only technique to pass in arguments
Arguments could be passed in and browse directly with sysargv, making it the best technique to pass in multiple arguments.