Coverage for src/robotide/contrib/testrunner/usages.py: 29%

4# Licensed under the Apache License, Version 2.0 (the "License");

5# you may not use this file except in compliance with the License.

6# You may obtain a copy of the License at

8# http://www.apache.org/licenses/LICENSE-2.0

10# Unless required by applicable law or agreed to in writing, software

11# distributed under the License is distributed on an "AS IS" BASIS,

12# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

13# See the License for the specific language governing permissions and

14# limitations under the License.

16try: (empty)

17 from robot.run import USAGE # First try the installed version (empty)

18except ImportError:

19 try:

20 from robotide.lib.robot.run import USAGE # Then try the provided version, 3.1.2

21 except ImportError:

22 USAGE = """

23Robot Framework -- A generic test automation framework

25Usage: robot|jybot|irobot [options] data_sources

26 or: python|jython|ipy -m robot.run [options] data_sources

27 or: python|jython|ipy path/to/robot/run.py [options] data_sources

28 or: java -jar robotframework.jar run [options] data_sources

30Robot Framework is a Python-based keyword-driven test automation framework for

31acceptance level testing and acceptance test-driven development (ATDD). It has

32an easy-to-use tabular syntax for creating test cases and its testing

33capabilities can be extended by test libraries implemented either with Python

34or Java. Users can also create new keywords from existing ones using the same

35simple syntax that is used for creating test cases.

37Depending is Robot Framework installed using Python, Jython, or IronPython

38interpreter, it has a start-up script, `robot`, `jybot` or `irobot`,

39respectively. Alternatively, it is possible to directly execute `robot.run`

40module (e.g. `python -m robot.run`) or `robot/run.py` script using a selected

41interpreter. Finally, there is also a standalone JAR distribution.

43Data sources given to Robot Framework are either test case files or directories

44containing them and/or other directories. Single test case file creates a test

45suite containing all the test cases in it and a directory containing test case

46files creates a higher level test suite with test case files or other

47directories as sub test suites. If multiple data sources are given, a virtual

48top level suite containing suites generated from given data sources is created.

50By default Robot Framework creates an XML output file and a log and a report in

51HTML format, but this can be configured using various options listed below.

52Outputs in HTML format are for human consumption and XML output for integration

53with other systems. XML outputs can also be combined and otherwise further

54processed with `rebot` tool. Run `rebot --help` for more information.

56Robot Framework is open source software released under Apache License 2.0. Its

57copyrights are owned and development supported by Nokia Solutions and Networks.

58For more information about the framework see https://robotframework.org/.

60Options

61=======

63 -N --name name Set the name of the top level test suite. Underscores

64 in the name are converted to spaces. Default name is

65 created from the name of the executed data source.

66 -D --doc documentation Set the documentation of the top level test suite.

67 Underscores in the documentation are converted to

68 spaces and it may also contain simple HTML formatting

69 (e.g. *bold* and https://url/).

70 -M --metadata name:value * Set metadata of the top level suite. Underscores

71 in the name and value are converted to spaces. Value

72 can contain same HTML formatting as --doc.

73 Example: --metadata version:1.2

74 -G --settag tag * Sets given tag(s) to all executed test cases.

75 -t --test name * Select test cases to run by name or long name. Name

76 is case and space insensitive and it can also be a

77 simple pattern where `*` matches anything and `?`

78 matches any char. If using `*` and `?` in the console

79 is problematic see --escape and --argumentfile.

80 -s --suite name * Select test suites to run by name. When this option

81 is used with --test, --include or --exclude, only

82 test cases in matching suites and also matching other

83 filtering criteria are selected. Name can be a simple

84 pattern similarly as with --test and it can contain

85 parent name separated with a dot. For example

86 `-s X.Y` selects suite `Y` only if its parent is `X`.

87 -i --include tag * Select test cases to run by tag. Similarly as name

88 with --test, tag is case and space insensitive and it

89 is possible to use patterns with `*` and `?` as

90 wildcards. Tags and patterns can also be combined

91 together with `AND`, `OR`, and `NOT` operators.

92 Examples: --include foo --include bar*

93 --include fooANDbar*

94 -e --exclude tag * Select test cases not to run by tag. These tests are

95 not run even if included with --include. Tags are

96 matched using the rules explained with --include.

97 -R --rerunfailed output Select failed tests from an earlier output file to be

98 re-executed. Equivalent to selecting same tests

99 individually using --test option.

100 --runfailed output Deprecated since RF 2.8.4. Use --rerunfailed instead.

101 -c --critical tag * Tests having given tag are considered critical. If no

102 critical tags are set, all tags are critical. Tags

103 can be given as a pattern like with --include.

104 -n --noncritical tag * Tests with given tag are not critical even if they

105 have a tag set with --critical. Tag can be a pattern.

106 -v --variable name:value * Set variables in the test data. Only scalar

107 variables with string value are supported and name is

108 given without `${}`. See --escape for how to use

109 special characters and --variablefile for a more

110 powerful variable setting mechanism.

111 Examples:

112 --variable str:Hello => ${str} = `Hello`

113 -v hi:Hi_World -E space:_ => ${hi} = `Hi World`

114 -v x: -v y:42 => ${x} = ``, ${y} = `42`

115 -V --variablefile path * Python or YAML file file to read variables from.

116 Possible arguments to the variable file can be given

117 after the path using colon or semicolon as separator.

118 Examples: --variablefile path/vars.yaml

119 --variablefile environment.py:testing

120 -d --outputdir dir Where to create output files. The default is the

121 directory where tests are run from and the given path

122 is considered relative to that unless it is absolute.

123 -o --output file XML output file. Given path, similarly as paths given

124 to --log, --report, --xunit, and --debugfile, is

125 relative to --outputdir unless given as an absolute

126 path. Other output files are created based on XML

127 output files after the test execution and XML outputs

128 can also be further processed with Rebot tool. Can be

129 disabled by giving a special value `NONE`. In this

130 case, also log and report are automatically disabled.

131 Default: output.xml

132 -l --log file HTML log file. Can be disabled by giving a special

133 value `NONE`. Default: log.html

134 Examples: `--log mylog.html`, `-l NONE`

135 -r --report file HTML report file. Can be disabled with `NONE`

136 similarly as --log. Default: report.html

137 -x --xunit file xUnit compatible result file. Not created unless this

138 option is specified.

139 --xunitskipnoncritical Mark non-critical tests on xUnit output as skipped.

140 -b --debugfile file Debug file written during execution. Not created

141 unless this option is specified.

142 -T --timestampoutputs When this option is used, timestamp in a format

143 `YYYYMMDD-hhmmss` is added to all generated output

144 files between their basename and extension. For

145 example `-T -o output.xml -r report.html -l none`

146 creates files like `output-20070503-154410.xml` and

147 `report-20070503-154410.html`.

148 --splitlog Split log file into smaller pieces that open in

149 browser transparently.

150 --logtitle title Title for the generated test log. The default title

151 is `<Name Of The Suite> Test Log`. Underscores in

152 the title are converted into spaces in all titles.

153 --reporttitle title Title for the generated test report. The default

154 title is `<Name Of The Suite> Test Report`.

155 --reportbackground colors Background colors to use in the report file.

156 Either `all_passed:critical_passed:failed` or

157 `passed:failed`. Both color names and codes work.

158 Examples: --reportbackground green:yellow:red

159 --reportbackground #00E:#E00

160 -L --loglevel level Threshold level for logging. Available levels: TRACE,

161 DEBUG, INFO (default), WARN, NONE (no logging). Use

162 syntax `LOGLEVEL:DEFAULT` to define the default

163 visible log level in log files.

164 Examples: --loglevel DEBUG

165 --loglevel DEBUG:INFO

166 --suitestatlevel level How many levels to show in `Statistics by Suite`

167 in log and report. By default all suite levels are

168 shown. Example: --suitestatlevel 3

169 --tagstatinclude tag * Include only matching tags in `Statistics by Tag`

170 and `Test Details` in log and report. By default all

171 tags set in test cases are shown. Given `tag` can

172 also be a simple pattern (see e.g. --test).

173 --tagstatexclude tag * Exclude matching tags from `Statistics by Tag` and

174 `Test Details`. This option can be used with

175 --tagstatinclude similarly as --exclude is used with

176 --include.

177 --tagstatcombine tags:name * Create combined statistics based on tags.

178 These statistics are added into `Statistics by Tag`

179 and matching tests into `Test Details`. If optional

180 `name` is not given, name of the combined tag is got

181 from the specified tags. Tags are combined using the

182 rules explained in --include.

183 Examples: --tagstatcombine requirement-*

184 --tagstatcombine tag1ANDtag2:My_name

185 --tagdoc pattern:doc * Add documentation to tags matching given pattern.

186 Documentation is shown in `Test Details` and also as

187 a tooltip in `Statistics by Tag`. Pattern can contain

188 characters `*` (matches anything) and `?` (matches

189 any char). Documentation can contain formatting

190 similarly as with --doc option.

191 Examples: --tagdoc mytag:My_documentation

192 --tagdoc regression:*See*_https://info.html

193 --tagdoc owner-*:Original_author

194 --tagstatlink pattern:link:title * Add external links into `Statistics by

195 Tag`. Pattern can contain characters `*` (matches

196 anything) and `?` (matches any char). Characters

197 matching to wildcard expressions can be used in link

198 and title with syntax %N, where N is index of the

199 match (starting from 1). In title underscores are

200 automatically converted to spaces.

201 Examples: --tagstatlink mytag:https://my.domain:Link

202 --tagstatlink bug-*:https://tracker/id=%1:Bug_Tracker

204 Remove keyword data from the generated log file.

205 Keywords containing warnings are not removed except

206 in `all` mode.

207 all: remove data from all keywords

208 passed: remove data only from keywords in passed

209 test cases and suites

210 for: remove passed iterations from for loops

211 wuks: remove all but the last failing keyword

212 inside `BuiltIn.Wait Until Keyword Succeeds`

213 name:<pattern>: remove data from keywords that match

214 the given pattern. The pattern is matched

215 against the full name of the keyword (e.g.

216 'MyLib.Keyword', 'resource.Second Keyword'),

217 is case, space, and underscore insensitive,

218 and may contain `*` and `?` as wildcards.

219 Examples: --removekeywords name:Lib.HugeKw

220 --removekeywords name:myresource.*

221 tag:<pattern>: remove data from keywords that match

222 the given pattern. Tags are case and space

223 insensitive and it is possible to use

224 patterns with `*` and `?` as wildcards.

225 Tags and patterns can also be combined

226 together with `AND`, `OR`, and `NOT`

227 operators.

228 Examples: --removekeywords foo

229 --removekeywords fooANDbar*

230 --flattenkeywords for|foritem|name:<pattern>|tag:<pattern> *

231 Flattens matching keywords in the generated log file.

232 Matching keywords get all log messages from their

233 child keywords and children are discarded otherwise.

234 for: flatten for loops fully

235 foritem: flatten individual for loop iterations

236 name:<pattern>: flatten matched keywords using same

237 matching rules as with

238 `--removekeywords name:<pattern>`

239 tag:<pattern>: flatten matched keywords using same

240 matching rules as with

241 `--removekeywords tag:<pattern>`

242 --listener class * A class for monitoring test execution. Gets

243 notifications e.g. when a test case starts and ends.

244 Arguments to the listener class can be given after

245 the name using colon or semicolon as a separator.

246 Examples: --listener MyListenerClass

247 --listener path/to/Listener.py:arg1:arg2

248 --warnonskippedfiles If this option is used, skipped test data files will

249 cause a warning that is visible in the console output

250 and the log file. By default skipped files only cause

251 an info level syslog message.

252 --nostatusrc Sets the return code to zero regardless of failures

253 in test cases. Error codes are returned normally.

254 --runemptysuite Executes tests also if the top level test suite is

255 empty. Useful e.g. with --include/--exclude when it

256 is not an error that no test matches the condition.

257 --dryrun Verifies test data and runs tests so that library

258 keywords are not executed.

259 --exitonfailure Stops test execution if any critical test fails.

260 --exitonerror Stops test execution if any error occurs when parsing

261 test data, importing libraries, and so on.

262 --skipteardownonexit Causes teardowns to be skipped if test execution is

263 stopped prematurely.

264 --randomize all|suites|tests|none Randomizes the test execution order.

265 all: randomizes both suites and tests

266 suites: randomizes suites

267 tests: randomizes tests

268 none: no randomization (default)

269 Use syntax `VALUE:SEED` to give a custom random seed.

270 The seed must be an integer.

271 Examples: --randomize all

272 --randomize tests:1234

273 --prerunmodifier class * Class to programmatically modify the test suite

274 structure before execution.

275 --prerebotmodifier class * Class to programmatically modify the result

276 model before creating reports and logs.

277 --console type How to report execution on the console.

278 verbose: report every suite and test (default)

279 dotted: only show `.` for passed test, `f` for

280 failed non-critical tests, and `F` for

281 failed critical tests

282 quiet: no output except for errors and warnings

283 none: no output whatsoever

284 -. --dotted Shortcut for `--console dotted`.

285 --quiet Shortcut for `--console quiet`.

286 -W --consolewidth chars Width of the console output. Default is 78.

287 -C --consolecolors auto|on|ansi|off Use colors on console output or not.

288 auto: use colors when output not redirected (default)

289 on: always use colors

290 ansi: like `on` but use ANSI colors also on Windows

291 off: disable colors altogether

292 Note that colors do not work with Jython on Windows.

293 -K --consolemarkers auto|on|off Show markers on the console when top level

294 keywords in a test case end. Values have same

295 semantics as with --consolecolors.

296 --monitorwidth chars Deprecated. Use --consolewidth instead.

297 --monitorcolors colors Deprecated. Use --consolecolors instead.

298 --monitormarkers value Deprecated. Use --consolemarkers instead.

299 -P --pythonpath path * Additional locations (directories, ZIPs, JARs) where

300 to search test libraries and other extensions when

301 they are imported. Multiple paths can be given by

302 separating them with a colon (`:`) or by using this

303 option several times. Given path can also be a glob

304 pattern matching multiple paths but then it normally

305 must be escaped or quoted.

306 Examples:

307 --pythonpath libs/

308 --pythonpath /opt/testlibs:mylibs.zip:yourlibs

309 -E star:STAR -P lib/STAR.jar -P mylib.jar

310 -E --escape what:with * Escape characters which are problematic in console.

311 `what` is the name of the character to escape and

312 `with` is the string to escape it with. Note that

313 all given arguments, incl. data sources, are escaped

314 so escape characters ought to be selected carefully.

315 Available escapes: amp (&), apos ('), at (@), bslash

316 (\\), colon (:), comma (,), curly1 ({), curly2 (}),

317 dollar ($), exclam (!), gt (>), hash (#), lt (<),

318 paren1 ((), paren2 ()), percent (%), pipe (|), quest

319 (?), quot ("), semic (;), slash (/), space ( ),

320 square1 ([), square2 (]), star (*)

321 Examples:

322 --escape space:_ --metadata X:Value_with_spaces

323 -E space:SP -E quot:Q -v var:QhelloSPworldQ

324 -A --argumentfile path * Text file to read more arguments from. Use special

325 path `STDIN` to read contents from the standard input

326 stream. File can have both options and data sources

327 one per line. Contents do not need to be escaped but

328 spaces in the beginning and end of lines are removed.

329 Empty lines and lines starting with a hash character

330 (#) are ignored.

331 Example file:

332 | --include regression

333 | --name Regression Tests

334 | # This is a comment line

335 | my_tests.html

336 | path/to/test/directory/

337 Examples:

338 --argumentfile argfile.txt --argumentfile STDIN

339 -h -? --help Print usage instructions.

340 --version Print version information.

341

342Options that are marked with an asterisk (*) can be specified multiple times.

343For example, `--test first --test third` selects test cases with name `first`

344and `third`. If an option accepts a value but is not marked with an asterisk,

345the last given value has precedence. For example, `--log A.html --log B.html`

346creates log file `B.html`. Options accepting no values can be disabled by

347using the same option again with `no` prefix added or dropped. The last option

348has precedence regardless of how many times options are used. For example,

349`--dryrun --dryrun --nodryrun --nostatusrc --statusrc` would not activate the

350dry-run mode and would return normal status rc.

351

352Long option format is case-insensitive. For example, --SuiteStatLevel is

353equivalent to but easier to read than --suitestatlevel. Long options can

354also be shortened as long as they are unique. For example, `--logti Title`

355works while `--lo log.html` does not because the former matches only --logtitle

356but the latter matches --log, --loglevel and --logtitle.

357

358Environment Variables

359=====================

360

361ROBOT_OPTIONS Space separated list of default options to be placed

362 in front of any explicit options on the command line.

363ROBOT_SYSLOG_FILE Path to a file where Robot Framework writes internal

364 information about parsing test case files and running

365 tests. Can be useful when debugging problems. If not

366 set, or set to special value `NONE`, writing to the

367 syslog file is disabled.

368ROBOT_SYSLOG_LEVEL Log level to use when writing to the syslog file.

369 Available levels are the same as for --loglevel

370 command line option and the default is INFO.

371

372Examples

373========

374

375# Simple test run with `robot` without options.

376$ robot tests.html

377

378# Using options and running with `jybot`.

379$ jybot --include smoke --name Smoke_Tests path/to/tests.txt

380

381# Executing `robot.run` module using Python.

382$ python -m robot.run --test test1 --test test2 test_directory

383

384# Running `robot/run.py` script with Jython.

385$ jython /path/to/robot/run.py tests.robot

386

387# Executing multiple test case files and using case-insensitive long options.

388$ robot --SuiteStatLevel 2 /my/tests/*.html /your/tests.html

389

390# Setting default options and syslog file before running tests.

391$ export ROBOT_OPTIONS="--critical regression --suitestatlevel 2"

392$ export ROBOT_SYSLOG_FILE=/tmp/syslog.txt

393$ robot tests.tsv

394"""