pipeline program
Component: cr-action:pipeline program (v1.0.2)
Added by: gfursin (2020-01-31 13:27:43)
Authors: Grigori Fursin
License: BSD 3-clause (code) and CC BY-SA 4.0 (data)
Source: GitHub
Creation date: 2015-01-28 08:47:33
CID: 081173242a88bc94:3bbc50d1eb87d47ecr-action:pipeline program  )

Sign up here to be notified when new results are reproduced or new CodeReef components are shared!

How to get and run stable version (under development):
  pip install codereef
  cr download module:program --version=1.0.2 --all
  ck pipeline program --help

How to get and run development version:
  pip install ck
  ck pull repo:ck-autotuning
  ck pipeline program --help

How to run from Python:
   import ck.kernel as ck

   r=ck.access({'action':'pipeline',
                'module_uoa':'program',
                ... See JSON API below ...
               })
   if r['return']>0: return r
   ...
Info about the CK module with this action: program
Workflow framework: CK
Development repository: ck-autotuning
Module description: program compilation and execution workflow (pipeline)
Workflow: program compilation, execution and validation pipeline (workflow)
API Python code: Link
JSON API:
        "    Input:  {
              (repo_uoa)             - program repo UOA
              (module_uoa)           - program module UOA
              (data_uoa)             - program data UOA
                 or
              (program_uoa)          - useful if univeral pipeline is used, i.e. ck run pipeline:program program_uoa=...
                 or taken from .cm/meta.json from current directory

              (random)               - if 'yes', random selection of program, cmd, dataset uoa and dataset file
                                       (to support collaborative program optimization)

              (skip_local)           - if 'yes', skip detection of program in a local path

              (program_tags)         - select programs by these tags

              (program_dir)          - force program directory

              (target)               - target machine added via 'ck add machine' with prepared target description
                                       (useful to create farms of machines for crowd-benchmarking and crowd-tuning using CK)

              (host_os)              - host OS (detect, if omitted)
              (target_os)            - OS module to check (if omitted, analyze host)
              (device_id)            - device id if remote (such as adb)

              (local_platform)       - if 'yes', use host_os/target_os from the current platform
                                       (useful when replaying experiments from another machine and even OS)

              (prepare)              - if 'yes', only prepare setup, but do not clean/compile/run program
              (save_to_file)         - if !='', save updated input/output (state) to this file
              (skip_interaction)     - if 'yes' and out=='con', skip interaction to choose parameters

              (skip_device_init)     - if 'yes', skip device init
              (skip_info_collection) - if 'yes', skip info collection

              (skip_device_info)     - if 'yes', skip any device info -
                                       useful to prepare experiment crowdsourcing packs for remote devices

                 Pipeline sections' settings:

              (compute_platform_id)  - if !='', set env['CK_COMPUTE_PLATFORM_ID']
              (compute_device_id)    - if !='', set env['CK_COMPUTE_DEVICE_ID']

              (no_platform_features)    - if 'yes', do not collect full platform features
              (no_dataset_features)     - if 'yes', do not search and extract data set features
              (no_clean)                - if 'yes', do not clean directory before compile/run
              (no_compile)              - if 'yes', do not compile program (useful when running the same program
                                           under different system state conditions: CPU/GPU freq, cache/bus contentions, etc)
              (compile_only_once)       - if 'yes', compile only at first iteration
              (no_compiler_description) - if 'yes', do not search for most close compiler description with flags ...
              (no_run)                  - if 'yes', do not run program
                                             useful when using autotuning to find bugs in compiler,
                                             or find differently generated code sequencies, etc ...
              (no_state_check)          - do not check system/CPU state (frequency) over iterations ...


              (generate_rnd_tmp_dir) - if 'yes', compile and run program in randomly generated temporal dir
                      or
              (tmp_dir)              - if !='', use this tmp_dir

              (skip_clean_after)     - if 'yes', do not remove run batch
              (keep)                 - the same as skip_clean_after

              (console)              - if 'yes', output from program goes to console rather than file
                                          (usually for testing/demos)

              (cmd_key)              - CMD key
              (cmd_keys)             - Select only from this list of available CMD keys

              (dataset_uoa)          - UOA of a dataset
              (dataset_file)         - dataset filename (if more than one inside one entry - suggest to have a UID in name)
              (extra_dataset_tags)   - list of extra data set tags (useful to set \"small\" during mobile phone crowdtuning)

              (compiler_env_uoa)     - env of a compiler

              (compile_type)         - static or dynamic (dynamic by default;
                                         however takes compiler default_compile_type into account)
                  or
              (static or dynamic)

              (compiler_description_uoa)    - compiler description UOA (module compiler),
                                              if not set, there will be an attempt to detect the most close
                                              by version

              (compiler_vars)        - dict with set up compiler flags (-Dvar=value) ->
                                       they will update the ones defined as default in program description ...

              (no_vars)              - skip compiler vars (if you want to use default ones from the sources) ...

              (remove_compiler_vars) - list of compiler vars to remove

              (extra_env_for_compilation) - set environment variables before compiling program

              (flags)                - compile flags
              (lflags)               - link flags

              (compiler_flags)       - dict from compiler description (during autotuning),
                                       if set, description should exist in input:choices_desc#compiler_flags# ...

              (best_base_flag)       - if 'yes', try to select best flag if available ...
              (speed)                - the same as above
              (skip_best_base_flag)  - if 'yes', do not use best base flag (useful for exploration of other levels -O2,-O1,etc)
              (env_speed)            - use environment flag for best optimization (CK_OPT_SPEED)

              (shared_solution_cid)  - CID-UID1-UID2 of the shared optimization solution at cKnowledge.org/repo
                                       You can find it by clicking on a \"Copy CID to clipboard\" button of a given solution.
                                       See example at http://cknowledge.org/repo/web.php?wcid=8289e0cf24346aa7:79bca2b76876b5c6
                                       27bc42ee449e880e:79bca2b76876b5c6-8289e0cf24346aa7-f49649288ab0accd

              (Ocid-uid1-uid2)         Substituting compiler -Ox levels with shared solutions in above format
                                       (-O27bc42ee449e880e:79bca2b76876b5c6-8289e0cf24346aa7-f49649288ab0accd)

              (select_best_base_flag_for_first_iteration) - if 'yes' and autotuning_iteration=0


              (env)                  - preset environment
              (env.{KEY})            - set env[KEY]=value (user-friendly interface via CMD)

              (deps.{KEY})           - set deps[KEY][\"uoa']=value (user-friendly interface via CMD to set any given dependency)
              (preset_deps)          - dict with {\"KEY\":\"UOA\"} to preset dependencies

              (params)               - dictionary with parameters passed via pre/post processing to third-party tools
                                       for example, to configure ARM Workload Automation
              (params.{KEY})         - set params[KEY]=value (user-friendly interface via CMD)

              (extra_env)            - extra environment as string
              (extra_run_cmd)        - extra CMD (can use $#key#$ for autotuning)
              (debug_run_cmd)        - substitute CMD with this one - usually useful for debugging to pre-set env for all deps
              (run_cmd_substitutes)  - dict with substs ($#key#$=value) in run CMD (useful for CMD autotuning)

              (sudo)                 - if 'yes', force using sudo
                                       (otherwise, can use ${CK_SUDO_INIT}, ${CK_SUDO_PRE}, ${CK_SUDO_POST})

              (affinity)             - set processor affinity for tihs program run (if supported by OS - see \"affinity\" in OS)
                                       examples: 0 ; 0,1 ; 0-3 ; 4-7  (the last two can be useful for ARM big.LITTLE arhictecture

              (repeat)               - repeat kernel via environment CT_REPEAT_MAIN if supported
              (do_not_reuse_repeat)  - if 'yes', do not reuse repeat across iterations - needed for dataset exploration, for example
              (skip_calibration)     - if 'yes', skip execution time calibration (otherwise, make it around 4.0 sec)
              (calibration_time)     - calibration time in string, 4.0 sec. by default
              (calibration_max)      - max number of iterations for calibration, 10 by default

              (statistical_repetition_number) - current statistical repetition of experiment
                                                (for example, may be used to skip compilation, if >0)
              (autotuning_iteration)          - (int) current autotuning iteration (automatically updated during pipeline tuning)
              (the_same_dataset)              - if 'yes', the dataset stays the same across iterations
                                                   so skip copying dataset to remote from 2nd iteration

              (repeat_compilation)     - if 'yes', force compilation, even if \"statistical_repetition_number\">0

              (cpu_freq)               - set CPU frequency, if supported (using SUDO, if also supported)
                                           using script ck-set-cpu-online-and-frequency
                                         if \"max\" - try to set to maximum using script ck-set-cpu-performance
                                         if \"min\" - try to set to minimum using scrupt ck-set-cpu-powersave

              (gpu_freq)               - set GPU frequency, if supported (using SUDO, if also supported)
                                           using script ck-set-gpu-online-and-frequency
                                         if \"max\" - try to set to maximum using script ck-set-gpu-performance
                                         if \"min\" - try to set to minimum using scrupt ck-set-gpu-powersave

              (energy)                  - if 'yes', start energy monitoring (if supported) using script ck-set-power-sensors

              (gprof)                   - if 'yes', use gprof to collect profile info
              (perf)                    - if 'yes', use perf to collect hardware counters
              (vtune)                   - if 'yes', use Intel vtune to collect hardware counters
              (sim)                     - if 'yes', use architecture simulator (found by tags \"arch\",\"sim\")
              (valgrind)                - if 'yes', add valgrind

              (dvdt_prof)               - if 'yes', run program under dividiti's OpenCL profiler
              (mali_hwc)                - if 'yes', attempt to extract MALI GPU hardware counters

              (milepost)                - if 'yes', attempt to extract static program features using Milepost GCC and cTuning CC
              (milepost_out_file)       - if !='', record extracted MILEPOST features to this JSON file

              (compile_timeout)         - (sec.) - kill compile job if too long
              (run_timeout)             - (sec.) - kill run job if too long

              (post_process_script_uoa) - run script from this UOA
              (post_process_subscript)  - subscript name
              (post_process_params)     - (string) add params to CMD

              (dependencies)         - compilation dependencies
              (deps_cache)           - cache with resolved dependencies for reuse (if needed)
              (reuse_deps)           - if 'yes' reuse dependencies
              (force_resolve_deps)   - if 'yes', force resolve deps (useful for crowd-tuning)

              (choices)              - exposed choices (if any)
              (choices_order)        - vector of flattened choices (useful if optimizations need order
                                        such as LLVM or using our GCC plugin iterface to reorder passes,
                                        since 'choices' dict does not have order)

              (features)             - exposed features (if any)

              (characteristics)      - measured characteristics/features/properties (if any)

              (state)                - kept across pipeline iterations (for example, during autotuning/exploration)

                                       (tmp_dir)    - if temporal directory is used, return it
                                                      (useful if randomly generated, to be reused for run or further iterations)
                                       (repeat)     - kernel repeat ...
                                       (features.platform.cpu) - CPU features/properties obtained during iterations
                                                                 to check that state didn't change ...

              (skip_output_validation)        - skip validation of output (dangerous during auto-tuning -
                                                  some optimizations may break semantics or change accuracy)
              (output_validation_repo)        - output validation repo UOA (when recording new output)
              (program_output_uoa)            - use this UOA to check/record program output 
                                                (to have the same output entry for groups of similar programs)

              (overwrite_reference_output)    - if 'yes', overwrite reference output (useful if broken)

              (quiet)                - if 'yes', automatically provide default answer to all questions when resolving dependencies ...

              (last_md5)             - if !='', check if MD5 and fail if didn't change!
              (last_md5_fail_text)   - to recognize that pipeline failure is not really a failure,
                                       but MD5 is the same (useful when pruning compiler flags found
                                       during collaborative autotuning, particularly via mobile devices
                                       (less time to prune results))

              (skip_print_timers)          - if 'yes', skip printing fine-grain timers after execution

              (add_rnd_extension_to_bin)   - if 'yes', add random extension to binary and record list
              (add_save_extension_to_bin)  - if 'yes', add '.save' to bin to save during cleaning ...

              (install_to_env)       - install dependencies to env instead of CK-TOOLS (to keep it clean)!

              (safe)                 - safe mode when searching packages first instead of detecting already installed soft
                                       (to have more deterministic build)

              (skip_exec)            - if 'yes', do not clean output files and skip exec to be able to continue
                                       post-processing during debuging
            }

    Output: {
              return       - return code =  0, if successful
                                         >  0, if error
              (error)      - error text if return > 0

              !!! The same as input, but with updated information !!!

              ready        - if 'yes', pipeline is ready (all obligatory choices are set)
                             if 'no', clean/compile/run program is postponed

              state        - should be preserved across autotuning, active (online) learning, exploration, validation iterations
            }

"
       


All versions:


Public comments

    Please log in to add your comment!


If you notice inapropriate content that should not be here, please report us as soon as possible and we will try to remove it within 48 hours!